INSIDE: LLMs' Internal States Retain the Power of Hallucination
  Detection (2402.03744)

🔍 Today's pick in Interpretability & Analysis of LMs: INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection by @chaochen et al.

Previous efforts in detecting hallucinations using model intrinsic information employed predictive uncertainty or self-consistency to detect evaluation. Authors contend that in these procedure the rich semantic information captured in model embeddings is inevitably lost while decoding tokens.

To prevent this information loss they propose EigenScore, an internal measure of responses’ self-consistency using the eigenvalues of sampled responses' covariance matrix in intermediate model layers to quantify answers’ diversity in the dense embedding space.

Results show that EigenScore outperforms logit-level methods for hallucination detection on QA tasks, especially when paired with inference time feature clipping to truncate extreme activations, reducing overconfident generations.

📄 Paper: https://huggingface.co/papers/2402.03744


Introducing Vision Arena (beta)! Based on the lmsys's ChatbotArena, we create a simple demo for testing different Vision LMs (VLMs). We now  support GPT-4V, Gemini-Pro-Vision, and Llava.  More updates and models will come soon! We are still in the development stage and for now and we'd love to hear your feedback and suggestions! Please help us vote for better VLMs in your own use cases here! :D  Kudos to Yujie Lu (UCSB)!
https://huggingface.co/spaces/WildVision/vision-arena

merve yuchenlin Dlbk victor osanseviero lewtun clefourrier multimodalart FunCube JustinLin610 VictorSanh drogozhang samusenps tdambrowitz AdinaY natolambert vishaal27 MoritzLaurer taesiri


We've been busy cooking up some interesting models at @jinaai, with a recent highlight being the release of our first batch of bilingual embedding models.

Internally labeled as `X+EN`, where X represents the target language and `EN` stays fixed, these models specialize in both monolingual tasks and cross-lingual retrieval tasks, crossing from X to EN.

You can find these models available on Huggingface:
1. German-English bilingual embedding: https://huggingface.co/jinaai/jina-embeddings-v2-base-de
2. Chinese-English bilingual embedding: https://huggingface.co/jinaai/jina-embeddings-v2-base-zh

We're also excited to announce that a Spanish bilingual embedding will be released in approximately two weeks.

Our evaluation across various MLM tasks has demonstrated that the Bilingual Backbone consistently outperforms state-of-the-art Multilingual Backbones like XLM-Roberta (given its focus on just two languages).

Despite being three times smaller than the leading multilingual models (e5-multilingual-large), our released bilingual embedding models have shown superior performance compared to e5-multilingual-large, excelling in both monolingual and cross-lingual search tasks.

Currently, we're putting the finishing touches on the technical report, which should be available on Arxiv by next week.

Looking ahead, the embedding team is gearing up for `jina-embeddings-v3`
 with some initial groundwork already underway. Stay tuned for more updates!

KnutJaegersberg merve Dlbk osanseviero tomaarsen FremyCompany victor


Yesterday we just released Qwen1.5. Maybe someday I can tell more about the experience. But this is is at least a good release even if it is not yet SOTA. There is not so many SOTA by the way. This time, we actually fixed a lot of problems.

1. Context lengths are finally unified for all sizes. Previously, a lot of users kept telling us that 14B only supports 2K (Yeah even dynamic NTK does not work that well and it can only be extended to around 4-5K. Let alone those know nothing about how to use dynamic NTK).

2. If you carefully use our base language models, you will find that they understand special tokens of ChatML, which means that you can directly use LoRA to train on data with ChatML format. Why you can't do this before? This is because if the base language model does not understand the special tokens, you need to make them trained, which means that you should turn on the training of embedding. This is disgusting and it often leads to problems when you use ZeRO3.

3. We did strengthen our base language models except for 72. You should find better base language models, especially for 7 and 14. Why not 72? Nah, hard to say, but will make it better.

4. About the multilingual capabilities. Yes we finally build up our multilingual evaluation system and find out that our new base language models have nice  performance in multilingual evaluation for base language models. This tells us that we should pay more attention to the post-training with multilingual data. And we did that too. This is why this time we tell you something about multilingual performance. It is for sure much much better than our models before this release.

5. Chat models are the most promising stuff. Before this release, we gave you the SFT models. But this time, we had very nice SFT+DPO models. Yeah not only annotators like them but also users like them. I am sure you developers will feel that way too.

s3nh osanseviero Pclanglais andysalerno hf-delta huodon KnutJaegersberg cin-hubert xu3kev hiyouga merve Stopwolf Dlbk ShushengYang lewtun Ji-Ha xianbao euclaise smerchi AdinaY Sangmin philschmid joaogante victor nateraw dlimeng bowiehsu jkeisling HDiffusion EditaZ

osanseviero merve xianbao AdinaY philschmid


DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open
  Language Models (2402.03300)

DeepSeekMath

Pushing the Limits of Mathematical Reasoning in Open Language Models

paper page: https://huggingface.co/papers/2402.03300

DeepSeekMath 7B has achieved an impressive score of 51.7% on the competition-level MATH benchmark without relying on external toolkits and voting techniques, approaching the performance level of Gemini-Ultra and GPT-4

clem ajibawa-2023 osanseviero merve Dlbk andreinigo distantquant hunkim alielfilali01 Sattineez mysticaltech


BARTScore: Evaluating Generated Text as Text Generation (2106.11520)

Understanding BARTScore 🛹

BARTScore is a text-generation evaluation metric that treats model evaluation as a text-generation task 🔄

Other metrics approach the evaluation problem from different ML task perspectives; for instance, ROUGE and BLUE formulate it as an unsupervised matching task, BLUERT and COMET as a supervised regression, and BEER as a supervised ranking task.

Meanwhile, BARTScore formulates it as a text-generation task. Its idea is to leverage BART's pre-trained contextual embeddings to return a score that measures either the faithfulness, precision, recall, or F-score response of the main text-generation model.

For example, if we want to measure faithfulness, the way it works is that we would take the source and the generated text from our model and use BART to calculate the log token probability of the generated text given the source; we can then weight those results and return the sum.

BARTScore correlates nicely with human scores, and it is relatively simple to implement.

📑 Here is the original BARTScore paper: https://huggingface.co/papers/2106.11520
🧑💻 And the GitHub repo to use this metric: https://github.com/neulab/BARTScore


Rethinking Interpretability in the Era of Large Language Models (2402.01761)

🔍 Today's pick in Interpretability & Analysis of LMs: Rethinking Interpretability in the Era of Large Language Models
by C. Singh, J. P. Inala, @mgalley, R. Caruana, @wyngjf

In this opinion piece, authors contend that the new capabilities of LLMs can deeply transform the scope of interpretability, moving from low-level explanations such as saliency maps to natural language explanations that would allow for natural interaction with users.

This ambitious goal is however hindered by LM’s natural tendency to hallucinate, their large size and their inherent opaqueness. Authors highlight in particular dataset explanations for knowledge discovery, explanations’ reliability and interactive explanations as important priorities for the future of interpretability research.

📄 Paper: https://huggingface.co/papers/2402.01761


Today my RunPod pod was broken and I didn't notice until I fully did setup it. So I have written the following tutorial for how to deploy a Pod and also verify it is not broken.

You can read on:

Patreon (public) : https://www.patreon.com/posts/how-to-deploy-on-97919576

Medium (public) : https://medium.com/@furkangozukara/how-to-deploy-a-pod-on-runpod-and-verify-it-is-working-20e47031c0b5

CivitAI (public) : https://civitai.com/articles/3994/how-to-deploy-a-pod-on-runpod-and-verify-it-is-working

LinkedIn (public) : https://www.linkedin.com/pulse/how-deploy-pod-runpod-verify-working-furkan-g%2525C3%2525B6z%2525C3%2525BCkara-lgplf%3FtrackingId=EuNOjpKCSQ%252BVfpiQV3D6KQ%253D%253D/?trackingId=EuNOjpKCSQ%2BVfpiQV3D6KQ%3D%3D

Dev . to (public) : https://dev.to/furkangozukara/how-to-deploy-a-pod-on-runpod-and-verify-it-is-working-3pop


StepCoder: Improve Code Generation with Reinforcement Learning from
  Compiler Feedback (2402.01391)

StepCoder

Improve Code Generation with Reinforcement Learning from Compiler Feedback

paper page: https://huggingface.co/papers/2402.01391

The advancement of large language models (LLMs) has significantly propelled the field of code generation. Previous work integrated reinforcement learning (RL) with compiler feedback for exploring the output space of LLMs to enhance code generation quality. However, the lengthy code generated by LLMs in response to complex human requirements makes RL exploration a challenge. Also, since the unit tests may not cover the complicated code, optimizing LLMs by using these unexecuted code snippets is ineffective. To tackle these challenges, we introduce StepCoder, a novel RL framework for code generation, consisting of two main components: CCCS addresses the exploration challenge by breaking the long sequences code generation task into a Curriculum of Code Completion Subtasks, while FGO only optimizes the model by masking the unexecuted code segments to provide Fine-Grained Optimization. In addition, we furthermore construct the APPS+ dataset for RL training, which is manually verified to ensure the correctness of unit tests. Experimental results show that our method improves the ability to explore the output space and outperforms state-of-the-art approaches in corresponding benchmarks.


Necessity is the mother of invention, and of Gradio components.

Sometimes we realize that we need a Gradio component to build a cool application and demo, so we just build it. For example, we just added a new gr.ParamViewer component because we needed it to display information about Python & JavaScript functions in our documentation.

Of course, our users should be able able to do the same thing for their machine learning applications, so that's why Gradio lets you build custom components, and publish them to the world 🔥

sbarman25 victor samusenps ysharma ajibawa-2023 gsarti osanseviero dvilasuero hmb notsahil


These past months, I've been busy baking a special sort of Croissant 🥐 with an awesome team !

🥐 CroissantLLM is a truly bilingual language model trained on 3 trillion tokens of French and English data. In its size category (<2B), it is the best model in French, but it also rivals the best monolingual English models !

💾 To train it, we collected, filtered and cleaned huge quantities of permissively licensed French data, across various domains (legal, administrative, cultural, scientific), and different text modalities (speech transcriptions, movie subtitles, encyclopedias, forums, webpages)...

⚖ Assessing  LLM performance is not easy, especially outside of English, and to this end we crafted a novel evaluation benchmark, FrenchBench, aiming to assess reasoning, factual knowledge, and linguistic capabilities of models in French !

🔎 The best current LLMs are hidden behind a shroud of mystery, trained with undisclosed training data mixes or strategies. We go the opposite way, releasing all of the project's artefacts (model checkpoints, data, training details, evaluation benchmarks...) We obtain 81 % of the Stanford FMTI transparency criterias, far ahead of even most open initiatives !

🧪Beyond a powerful industrial resource, our transparent initiative is a stepping stone for many scientific questions ! How does teaching a model two languages instead of one splits its monolingual ability ? Does training on so much French help the model integrate French-centric knowledge and cultural biases ? How does the model memorize the training data ?

Many more things to say, for those interested, I recommend checking out:

🗞 The blogpost: https://huggingface.co/blog/manu/croissant-llm-blog
📖 The 45 page report with lots of gems: https://arxiv.org/abs/2402.00786
🤖 Models, Data, Demo: https://huggingface.co/croissantllm

eliolio gsarti euclaise eliebak kramp yozozaya clem blanchon victor mvaloatto osanseviero samusenps santiviquez ksiabani batmac nouamanetazi davanstrien fffiloni alielfilali01 sbrandeis Soubz velaia


Looking for a Needle in a Haystack: A Comprehensive Study of
  Hallucinations in Neural Machine Translation (2208.05309)

SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for
  Generative Large Language Models (2303.08896)

LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain
  Conversations with Large Language Models (2305.13711)

Some of my results from experimenting with hallucination detection techniques for LLMs 🫨🔍

First, the two main ideas used in the experiments—using token probabilities and LLM-Eval scores—are taken from these three papers:

1. https://huggingface.co/papers/2208.05309
2. https://huggingface.co/papers/2303.08896
3. https://huggingface.co/papers/2305.13711

In the first two, the authors claim that computing the average of the sentence-level token probabilities is the best heuristic for detecting hallucinations. And from my results, we do see a weak positive correlation between average token probabilities and ground truth. 🤔

The nice thing about this method is that it comes with almost no implementation cost since we only need the output token probabilities from the generated text, so it is straightforward to implement.

The third paper proposes an evaluation shema where we do an extra call to an LLM and kindly ask it to rate on a scale from 0 to 5 how good the generated text is on a set of different criteria. 📝🤖

I was able to reproduce similar results to those in the paper. There is a moderate positive correlation between the ground truth scores and the ones produced by the LLM.

Of course, this method is much more expensive since we would need one extra call to the LLM for every prediction that we would like to evaluate, and it is also very sensitive to prompt engineering. 🤷

osanseviero victor clem samusenps gsarti KasperNomm JasperV13 sbrandeis


A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for
  Verifiers of Reasoning Chains (2402.00559)

🔍 Today's pick in Interpretability & Analysis of LMs: A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains by @alonjacovi @yonatanbitton B. Bohnet J. Herzig @orhonovic M. Tseng M. Collins @roeeaharoni @mega

This work introduces a new methodology for human verification of reasoning chains and adopts it to annotate a dataset of chain-of-thought reasoning chains produced by 3 LMs. The annotated dataset, REVEAL, can be used to benchmark automatic verifiers of reasoning in LMs.

In their analysis, the authors find that LM-produced CoTs generally contain faulty steps, often leading to incorrect automatic verification. In particular, CoT-generating LMs are found to produce non-attributable reasoning steps often, and reasoning verifiers generally struggle to verify logical correctness.

📄 Paper: https://huggingface.co/papers/2402.00559
🔡 Dataset: https://huggingface.co/datasets/google/reveal

s3nh roeeaharoni alonjacovi osanseviero clem manu


📣 DPO Dutch model release + datasets

After teasing for a while, I am finally releasing **GEITje 7B Ultra**, building upon the great GEITje 7B by @Rijgersberg. New contributions include: large new datasets for SFT (instruction/chat), two datasets for DPO training (i.e. RLAIF), and an SFT and DPO version of GEITje. The READMEs describe everything well (I hope), and I'll also share more info on social medias tomorrow.

For me this is a huge release, the datasets more so than the models. I'm especially pleased with UltraChat, which I created with the intent of having a diverse dataset - the model must be able to communicate with different types of users. So the user questions are created as if they were written by different personas, e.g. language learners, young children, experts, critics, etc. The focus with this is "building a good communication bot that is accessible and can handle different kinds of user input".

I wish I could find the time to also write a paper to get some "academic recognition" but that'll have to wait for now. I just want to bring it to the public so that others can play with it and use it to build new, cool stuff!

I hope that you can all appreciate the work. Let's build some cool stuff with it!

Models:
- Demo: https://huggingface.co/spaces/BramVanroy/GEITje-7B-ultra
- DPO Model: https://huggingface.co/BramVanroy/GEITje-7B-ultra
- SFT model (not recommended): https://huggingface.co/BramVanroy/GEITje-7B-ultra-sft

Datasets with GPT-4 turbo completions:
  - No robots (~10k instructions): https://huggingface.co/datasets/BramVanroy/no_robots_dutch
  - UltraChat (~200k instructions): https://huggingface.co/datasets/BramVanroy/ultrachat_200k_dutch
  - UltraFeedback (DPO with GPT4+GEITje chat, ~50k): https://huggingface.co/datasets/BramVanroy/ultra_feedback_dutch
  - Orca DPO Pairs  (DPO with GPT4+GEITje chat, ~10k): https://huggingface.co/datasets/BramVanroy/orca_dpo_pairs_dutch

samusenps osanseviero dragonkue beomi p208p2002 gsarti s3nh taufiqdp victor Stopwolf Robbert jvdgoltz seostar clem cast42 jvh ajrogier

samusenps osanseviero gsarti HuggyMonkey


Introducing model-similarities, a new simple tool to contrast two models

A straightforward yet insightful tool designed to shed light on the similarities between various models. Discover it now at [Model Similarity GitHub Repository](https://github.com/fblgit/model-similarity).

This project is in its nascent stages, and we're eager for contributions and enhancements. Crafted with simplicity at its core, the tool performs two primary comparisons:
- Weight similarities, utilizing a simple approach to contrast vector differences (A != B).
- Cosine similarity between the parameters of models A and B, providing a nuanced measure of their alignment.

Included in the repository are sample analyses and reports that validate model card claims, particularly regarding the training specifics of transformer components such as MLP, Attention, etc. Remarkably, these samples reveal 100% similarity scores between those parts of the models, pinpointing the exact base model utilized.

Join us in refining and expanding this tool. Whether you're looking to contribute code, ideas, or both, your input will help transform this into a resource for everyone.

fblgit shivamanbhule thomasgauthier osanseviero distantquant victor samusenps shuvom mohammedbriman santiviquez clem alielfilali01


🙋🏻♂Hey there folks ,

i'm so impressed by the reception that https://huggingface.co/collabora/whisperspeech has recieved !

Check out the cool demo here :  https://huggingface.co/spaces/collabora/WhisperSpeech

- Open issue : how do we provide MPS support ? cc. @intel :-) looking into this now , any leads welcome!

check out also [collabora/whisperfusion](https://github.com/collabora/WhisperFusion)

hope you enjoy ! 🤗

blanchon samusenps matlok tollefj osanseviero victor clem


🚨📢🚀 Introducing Hercules-v2.0! A robust, multifaceted dataset for advanced models to excel in specialized domains. 🔬🌌📚🚀

📈 1.3M examples from sources derived from OpenHermes-2.5, covering Biology, Physics, Math, CS, Instruction Following, Function Calling, and Roleplay.

🔬 Enhance natural language understanding and processing in diverse domains.

🚀 Develop models for complex instructions, function calls, and roleplay scenarios.

📄 Licensed under Apache-2.0.

Thank you to all contributors and OpenHermes-2.5 creator! 🎉

Check it out here: https://huggingface.co/datasets/Locutusque/hercules-v2.0

📣 Update: After fine-tuning Mistral 7B on 100,000 examples of Hercules-v2.0, it earns an average score of 62 on Open LLM Leaderboard, outperforming OpenHermes-2.5 and OpenChat-3.5. 🎉

Check out this model here: https://huggingface.co/Locutusque/Hercules-2.0-Mistral-7B

taufiqdp osanseviero Tonic samusenps victor Locutusque kramp Severian clem bunnycore sethuiyer statler


GPU Poor POV: Burnout

Sometimes we do not have an energy to post about AI and new methods.
And thats totally ok, I guess.
Remember to sleep well and drink a lot of water. Have a great day :D <3

vicgalle Epiculous Ji-Ha samusenps visheratin MaziyarPanahi osanseviero joey00072 not-lain saishf victor Meggido PerceptiveFocusInc mvaloatto antiven0m taufiqdp kramp ajibawa-2023 award40 s3nh lusstta alielfilali01 Tonic mertbozkir KnutJaegersberg vincentweisser sbrandeis AtAndDev


ICYMI! Nomic Embed, the first fully open long context text embedder to beat OpenAI

- Open source, open weights, open data
- Beats OpenAI text-embeding-3-small and Ada on short and long context benchmarks
- Day 1 integrations with Langchain, LlamaIndex, MongoDB, and Sentence Transformers

Check out https://huggingface.co/nomic-ai/nomic-embed-text-v1 for the model weights.

Technical report: https://static.nomic.ai/reports/2024_Nomic_Embed_Text_Technical_Report.pdf
Blog Post: https://blog.nomic.ai/posts/nomic-embed-text-v1
Original Tweet Thread: https://x.com/nomic_ai/status/1753082063048040829?s=20

bstadt osanseviero davanstrien clem jeffboudier yjernite MaziyarPanahi julien-c SwimminPizza dave3991 samusenps baitao666 VictorSanh radames blanchon Rexhaif Tonic

osanseviero davanstrien clem yjernite julien-c VictorSanh radames


Lots of cool Gradio custom components, but is the most generally useful one I've seen so far: insert a Modal into any Gradio app by using the `modal` component!

```py
from gradio_modal import Modal

with gr.Blocks() as demo:
    gr.Markdown("### Main Page")
    gr.Textbox("lorem ipsum " * 1000, lines=10)

    with Modal(visible=True) as modal:
        gr.Markdown("# License Agreement")
```

victor s3nh sbarman25 eskayML akhaliq RSHVR dwipper

MaziyarPanahi afrideva samusenps akhaliq AseidasVera RSHVR


I'm happy to announce that ✨ Image to Music v2 ✨ is ready for you to try and i hope you'll like it too ! 😌

This new version has been crafted with transparency in mind,
so you can understand the process of translating an image to a musical equivalent.

How does it works under the hood ? 🤔

First, we get a very literal caption from https://huggingface.co/microsoft/kosmos-2-patch14-224; this caption is then given to a LLM Agent (currently https://huggingface.co/HuggingFaceH4/zephyr-7b-beta )which task is to translate the image caption to a musical and inspirational prompt for the next step.

Once we got a nice musical text from the LLM, we can send it to the text-to-music model of your choice:
MAGNet, MusicGen, AudioLDM-2, Riffusion or Mustango

Instead of the previous version of Image to Music which used Mubert API, and could output curious and obscure combinations, we only provide open sourced models available on the hub, called via the gradio API.

Also i guess the music result should be more accurate to the atmosphere of the image input, thanks to the LLM Agent step.

Pro tip, you can adjust the inspirational prompt to match your expectations, according to the chosen model and specific behavior of each one 👌

Try it, explore different models and tell me which one is your favorite 🤗
—› https://huggingface.co/spaces/fffiloni/image-to-music-v2

victor samusenps osanseviero phanes abidlabs koheingt adhisetiawan Indomitable-ai boqsc deepkyu mvaloatto MaziyarPanahi powzone kelsy Chief-Inspector pe65374 Swhalen98 aceeee Aixbox

naorex ivied7 MehdiLeZ eisneim OmbelineM


First post alert 🚨

Super excited to share with you my first Chat assistant :

`HuggingAssist`, meant to offer guidance with the large HuggingFace ecosystem

Chat with it from here : https://hf.co/chat/assistant/65bd0adc08560e58be454d86

It would be more helpful when the RAG / WEB features are available !
Looking forward to it 🔥

ps : tnx @Chunte for the cool Huggies

osanseviero nsarrazin victor EmilyWitko clem pierrci abidlabs samusenps BrigitteTousi KBayoud antmaiorino Tohito kramp mammour


ReAGent: Towards A Model-agnostic Feature Attribution Method for
  Generative Language Models (2402.00794)

🔍 Today's pick in Interpretability & Analysis of LMs: ReAGent: Towards A Model-agnostic Feature Attribution Method for Generative Language Models  by @casszhao and B. Shan

Authors propose Recursive Attribution Generation (ReAGent), a perturbation-based feature attribution approach specifically conceived for generative LMs. The method employs a lightweight encoder LM to replace sampled input spans with valid alternatives and measure the effect of the perturbation on the drop in next token probability predictions. ReAGent is shown to consistentlyoutperform other established approaches across several models and generation tasks in terms of token- and sentence-level faithfulness.

📄 Paper: https://huggingface.co/papers/2402.00794
💻 Code: https://github.com/casszhao/ReAGent


🔥 New LLM leaderboard on the hub: NPHardEval!

It uses questions of logic, of different mathematical complexities, as a proxy for reasoning abilities. It notably removes questions relying on arithmetic, to really focus on logical abilities.
What's interesting imo is the potential to really study a model performance at different levels of complexity.

Bonus: Since the questions can be generated automatically, it's going to be dynamic, updated monthly! 🚀
https://huggingface.co/spaces/NPHardEval/NPHardEval-leaderboard

Read more about how their questions are generated in the intro blog: https://huggingface.co/blog/leaderboards-on-the-hub-nphardeval

Congrats to @lizhouf, @wenyueH, @hyfrankl and their teams!

alvarobartt osanseviero alielfilali01 samusenps lizhouf diwank hyfrankl kramp joaogante hivaze Kukedlc sbrandeis


OLMo: Accelerating the Science of Language Models (2402.00838)

OLMo

Accelerating the Science of Language Models

paper page: https://huggingface.co/papers/2402.00838

a state-of-the-art, truly Open Language Model and its framework to build and study the science of language modeling. Unlike most prior efforts that have only released model weights and inference code, we release OLMo and the whole framework, including training data and training and evaluation code. We hope this release will empower and strengthen the open research community and inspire a new wave of innovation.

vladbogo Dlbk osanseviero samusenps clem taufiqdp mathiasn1 blanchon matlok Ji-Ha Iliassti nold Keesh SX-GitHub


🔥 New on HuggingChat: Assistants!

Today we are releasing Assistants on HuggingChat!
Assistants are a fun way to package your prompts and share them with the world - powered by Open source Models of course!

Learn more about Assistants here: https://huggingface.co/spaces/huggingchat/chat-ui/discussions/357
Browse Assistants here: https://huggingface.co/chat/assistants

hmb dtripathi Dlbk Sylvestre AmethystAmenity yasserrmd osanseviero andrewrreed chkla alielfilali01 kramp samusenps clem taufiqdp adamelliotfields blanchon beomi marianbasti Nymbo

PierreLepagnol samusenps neovalle clem fatemex blanchon beomi Nymbo


If you're looking for geospatial datasets, you might find what you're looking for in the Geospatial Datasets Collections:
https://huggingface.co/collections/blanchon/geospatial-datasets-656ddb097c934a7b3c4d4619

s3nh osanseviero clem samusenps doof-ferb severo


GPU Poor POV: Quantization

Today I want to share with you my notebook plug and play code
which help me a lot through my quantization journey.
Hope youll find it interesting it could be a good starter point to
gguf some of your awesome models :)

Have a great day <3

https://s3nh.bearblog.dev/gpu-poor-pov-gguf-snippet/

lunarflu Aspik101 clem naveedtrainer osanseviero samusenps Csplk taufiqdp neovalle Tanvir1337 Ji-Ha jtatman AtAndDev

lunarflu clem naveedtrainer impactframes Tanvir1337 alielfilali01 s3nh Ji-Ha AtAndDev

lunarflu clem naveedtrainer osanseviero Tanvir1337 Ji-Ha AtAndDev


Just out: new custom Gradio component specifically designed for code completion models 🔥

hmb dhuynh95 sbarman25 julien-c osanseviero victor lunarflu clem yehiaserag samusenps aust-t alielfilali01 gsarti ThreadAbort not-lain calmgoose deepkyu freddyaboulton


release day release day! OLMo 1b + 7b out today 🥳 and 65b coming soon...

With OLMo, we are really focused on advancing the study of LLMs. We release **everything**, from toolkit to create its training dataset (dolma) to training & inference code:

- OLMo paper: https://allenai.org/olmo/olmo-paper.pdf
- OLMo train code: https://github.com/allenai/OLMo
- OLMo eval code: https://github.com/allenai/OLMo-Eval
- OLMo 7b: https://huggingface.co/allenai/OLMo-7B
- OLMo 1b: https://huggingface.co/allenai/OLMo-1B
- Dolma paper: https://allenai.org/olmo/dolma-paper.pdf
- Dolma dataset v1.6: https://huggingface.co/datasets/allenai/dolma
- Dolma toolkit v1.0:  https://github.com/allenai/dolma

osanseviero dim julien-c andysalerno juancopi81 lunarflu clem davanstrien mdouglas samusenps kramp yizhongw beomi alielfilali01 mathiasn1 adamelliotfields vbuhoijymzoi Tonioesparza loubnabnl dashfunnydashdash


Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion
  Tokens (2401.17377)

Infini-gram

Scaling Unbounded n-gram Language Models to a Trillion Tokens

demo: https://huggingface.co/spaces/liujch1998/infini-gram

paper page: https://huggingface.co/papers/2401.17377

train them at the same data scale as neural LLMs -- 1.4 trillion tokens. This is the largest n-gram model ever built. Second, existing n-gram models use small n which hinders their performance; we instead allow n to be arbitrarily large, by introducing a new infty-gram LM with backoff. Instead of pre-computing n-gram count tables (which would be very expensive), we develop an engine named infini-gram -- powered by suffix arrays -- that can compute infty-gram (as well as n-gram with arbitrary n) probabilities with millisecond-level latency. The infty-gram framework and infini-gram engine enable us to conduct many novel and interesting analyses of human-written and machine-generated text: we find that the infty-gram LM has fairly high accuracy for next-token prediction (47%), and can complement neural LLMs to greatly reduce their language modeling perplexities. When analyzing machine-generated text, we also observe irregularities in the machine--infty-gram agreement level with respect to the suffix length, which indicates deficiencies in neural LLM pretraining and the positional embeddings of Transformers. We open-source our infini-gram engine in the hopes of enabling more study on how to best use verbatim information retrieved from large text corpora.

sankeerth1729 Maani lunarflu clem osanseviero soldni sacharbit matlok


Vision LLM for #edgecomputing ?

@openbmb, who OS'ed the UltraFeedback dataset before, released a series of strong eco-friendly yet powerful LLMs

- MiniCPM: 2B model that competes with Mistral-7B

- MiniCPM-V: 3B vision LLM on edge!
- MiniCPM-V: 3B vision LLM on edge!


H2O-Danube-1.8B Technical Report (2401.16818)

Happy to share H2O-Danube-1.8b, a small 1.8b model based trained on only 1T natural language tokens showing competitive metrics across benchmarks in the <2B model space.

Base weights: https://huggingface.co/h2oai/h2o-danube-1.8b-base
Chat weights: https://huggingface.co/h2oai/h2o-danube-1.8b-chat
Technical report: https://huggingface.co/papers/2401.16818

osanseviero Maani lunarflu mdouglas stefan-jo


It seems February started with a fully open source AI renaissance 🌟

Models released with fully open dataset, training code, weights ✅

LLM - https://huggingface.co/collections/allenai/olmo-suite-65aeaae8fe5b6b2122b46778 🧠
Embedding - https://huggingface.co/nomic-ai/nomic-embed-text-v1 📚 (sota!)

And it's literally February 1st - can't wait to see what else the community will bring 👀

clem multimodalart osanseviero BrigitteTousi Maani lunarflu leigh301 dillfrescott zamply alielfilali01 impactframes MdnghtToker OmbelineM

fffiloni multimodalart BrigitteTousi lunarflu clem greencookies dillfrescott


Today, we’re releasing our first pretrained Open Language Models (OLMo) at the Allen Institute for AI (AI2), a set of 7 billion parameter models and one 1 billion parameter variant. This line of work was probably the main reason I joined AI2 and is the biggest lever I see possible to enact meaningful change in how AI is used, studied, and discussed in the short term.

Links at the top because that's what you want:
* Core 7B model: https://huggingface.co/allenai/OLMo-7B
* 7B model twin (different GPU hardware): https://huggingface.co/allenai/OLMo-7B-Twin-2T
* 1B model: https://huggingface.co/allenai/OLMo-1B
* Dataset: https://huggingface.co/datasets/allenai/dolma
* Paper (arxiv soon): https://allenai.org/olmo/olmo-paper.pdf
* My personal blog post: https://www.interconnects.ai/p/olmo

OLMo will represent a new type of LLM enabling new approaches to ML research and deployment, because on a key axis of openness, OLMo represents something entirely different. OLMo is built for scientists to be able to develop research directions at every point in the development process and execute on them, which was previously not available due to incomplete information and tools.

Depending on the evaluation methods, OLMo 1 is either the best 7 billion parameter base model available for download or one of the best. This relies on a new way of thinking where models are judged on parameter plus token budget, similar to how scaling laws are measured for LLMs.

We're just getting started, so please help us learn how to be more scientific with LLMs!

julien-c osanseviero thomwolf AdinaY clem DnzzL LZDXN PereLluis13 venkateshmurugadas 90Barricade93 linoyts rbiswasfc Hamish victor yleo BrigitteTousi ArunkumarVR samforeman Maani lunarflu davanstrien socialprescribing kramp beomi alielfilali01 jeffboudier Tanvir1337 patrickocal Gilojond thewise eswardivi anileo1 shtoshni ahkarami radames

thomwolf clem fffiloni victor BrigitteTousi lunarflu socialprescribing beomi ahkarami radames

osanseviero clem victor BrigitteTousi lunarflu beomi alielfilali01


Had a lot of fun making this plot today.

If someone ever asks you why you need ML monitoring, show them this picture 😂


In-Context Language Learning: Architectures and Algorithms (2401.12973)

🔍 Today's pick in Interpretability & Analysis of LMs: Gradient-Based Language Model Red Teaming  by N. Wichers, C. Denison and @beirami

This work proposes Gradient-Based Red Teaming (GBRT), a red teaming method for automatically generating diverse prompts inducing an LM to output unsafe responses.

In practice, prompts are learned by scoring LM responses with a safety-trained probing classifier, and back-propagating through frozen classifier and LM to update the prompt.

Authors experiment with variants of GBRT aimed at inducing realistic prompts in an efficient way, and GBRT prompts are more likely to generate unsafe responses than those found by established RL-based red teaming methods. Moreover, these attacks are shown to succeed even when the LM has been fine-tuned to produce safer outputs.

📄 Paper: https://huggingface.co/papers/2401.12973
💻 Code: https://github.com/google-research/google-research/tree/master/gbrt

clem osanseviero lunarflu beirami 9voltfan2009


Let me introduce you LLE:  Leaks, leaks everywhere!

A quick experiment I've carried out on around 600 datasets from the HF Hub, the results are stored in https://huggingface.co/datasets/lbourdois/LLE, and the methodology is described in
https://huggingface.co/blog/lbourdois/lle

tomaarsen Dlbk vuluu clem osanseviero dhuynh95 alielfilali01 gsarti


Introducing Gajendra!

An early release of our 7B Hindi-Hinglish-English Instruction fine-tuned language model.

Model: https://huggingface.co/BhabhaAI/Gajendra-v0.1

We additionally explore ways to filter examples that can be translated from English to Hindi and are releasing initial versions of both dataset and model for it.

Model: https://huggingface.co/BhabhaAI/Mistral-translation-classify
Dataset: https://huggingface.co/datasets/BhabhaAI/translation-classify

Looking forward to collaborate with open source community to accelerate and release Hindi LLMs.

samusenps VanshGehlot victor osanseviero julien-c sbarman25 satpalsr alielfilali01 smangrul


What is WhisperKit?
WhisperKit is a pure Swift package built by @argmaxinc  that enables real-time Whisper inference on iPhone, Mac and even Apple Watch in two lines of code. It is built on Core ML and can be configured to run on the Neural Engine or GPU or both. WhisperKit models are published and distributed through the Hugging Face Hub. Anyone can publish a Whisper fine-tune and evaluate it on their own dataset using WhisperKit!

WhisperKit models: https://huggingface.co/argmaxinc/whisperkit-coreml
Example App: https://testflight.apple.com/join/LPVOyJZW
Swift Package: https://github.com/argmaxinc/WhisperKit
Python toolkit: https://github.com/argmaxinc/whisperkittools

osanseviero victor VanshGehlot C00100011 fffiloni ZachNagengast dillfrescott Araeynn

samusenps vishalvis ZachNagengast dillfrescott Araeynn pcuenq


Introducing the "UltraTextbooks" dataset 🚀📚
Check it out here: https://huggingface.co/datasets/Locutusque/UltraTextbooks
📘 A comprehensive collection of high-quality synthetic and human-written textbooks
👨🎓 Spanning various subjects and programming languages
🔧 Designed for advanced NLP tasks like language modeling, educational QA, text summarization, and content generation for edu purposes
🚀 Future expansions planned with additional data sources to enhance the corpus
👇 Data composition highlights 👇
- Blend of synthetic and human-written material
- Includes topics from general edu to specialized areas
- Structured with field "text"
🧩 Data collection from various Hugging Face datasets, guided by a diverse and comprehensive curation rationale
🚧 Limitations may exist, so report any issues you encounter

samusenps Felladrin osanseviero vicgalle VanshGehlot dhuynh95 Locutusque aloobun afrideva Lee1990

pszemraj Pelmeshek osanseviero Locutusque qinluo Lee1990


moondream1 can now be used directly from transformers!

Felladrin osanseviero clem cnmoro maywell samusenps Locutusque VanshGehlot fffiloni not-lain Sylvestre levisz lrzjason ajibawa-2023 Aptronym

YaTharThShaRma999 sujitvasanth osanseviero talgaurdian clem taufiqdp not-lain piebro impactframes Nymbo


📣 NEW on HF

the Dataset Viewer is now available on *private datasets* too

You need to be a PRO or a Enterprise Hub user. 🔥

Great work from our Datasets team 🥰: @lhoestq @severo @polinaeterna @asoria @albertvillanova and the whole team 🥰

lhoestq severo clem blanchon osanseviero pierrci albertvillanova VanshGehlot vladbogo AdinaY BrigitteTousi michaljunczyk not-lain Nymbo

fffiloni clem osanseviero Pelmeshek kramp mvaloatto awacke1 AdinaY BrigitteTousi Tonic not-lain Nymbo


🔥 New LLM leaderboard on the hub: an Enterprise Scenarios Leaderboard!

This work evaluates LLMs on several real world use cases (Finance documents, Legal confidentiality, Customer support, ...), which makes it grounded, and interesting  for companies! 🏢
Bonus: the test set is private, so it's hard to game 🔥
https://huggingface.co/spaces/PatronusAI/enterprise_scenarios_leaderboard

Side note: I discovered through this benchmark that you could evaluate "Engagingness" of an LLM, which could also be interesting for our LLM fine-tuning community out there.

Read more about their different tasks and metrics in the intro blog: https://huggingface.co/blog/leaderboards-on-the-hub-patronus

Congrats to @sunitha98 who led the leaderboard implementation, and to @rebeccaqian and @anandnk24, all at Patronus AI !


InstantID-2V is out ! ✨

It's like InstantID, but you get a video instead. Nothing crazy here, it's simply a shortcut between two demos.

Let's see how it does work with gradio API:

1. We call https://huggingface.co/spaces/InstantX/InstantID with a conditional pose from cinematic camera shot (example provided in the demo)
2. Then we send the previous generated image to https://huggingface.co/ali-vilab/i2vgen-xl

Et voilà 🤗 Try it : https://huggingface.co/spaces/fffiloni/InstantID-2V

—
Note that generation can be quite long, so take the opportunity to brew you some coffee 😌
If you want to skip the queue, you can of course reproduce this pipeline manually

linoyts akhaliq yang100 phanes PeepDaSlan9 samusenps victor osanseviero clem echobingo147 AdinaY cansa danielus


🚀 The Open Source AI community needs more open datasets for improving Open LLMs. Excited to share our new open dataset for boosting chat models:

🎉 Welcome Distilabel Capybara DPO, a multi-turn, high-quality preference dataset.

https://huggingface.co/datasets/argilla/distilabel-capybara-dpo-7k-binarized

Why?
Best closed chat models are built on top of multi-turn dialogue preference data. The OSS community lacks these datasets. This dataset is the first in the series to close this gap.

Is this dataset useful?
To test this dataset, we've built our virtual launching partner:

🎉 Welcome CapybaraHermes, a preference tuned OpenHermes with increased second turn capabilities on MTBench

https://huggingface.co/argilla/CapybaraHermes-2.5-Mistral-7B

As usual, models are the least important to us. We like to focus on the data. Our mission is to build and share high-quality datasets, sharing our methods in the open so the community can improve upon them.

That's why, we took some time to describe the full methodology on the dataset card, check it out and give us feedback! Data and methods are never perfect!

Finally, this is just a preview version and would love to collaborate with you to add more benchmarking results, what hyperparams work for DPO'ing models, what mix of datasets, etc.

Expect some more datasets in the coming weeks. Let's build the best data for AI, together.

davanstrien lhoestq osanseviero victor severo julien-c samusenps clem Smorty100 Arjun4707 gabrielmbmb kramp mayurninama davidberenstein1957 hfthorvaldur leonardlin alvarobartt alielfilali01 sbrandeis


Weaver: Foundation Models for Creative Writing (2401.17268)

Weaver: Foundation Models for Creative Writing

https://huggingface.co/papers/2401.17268

most-capable Weaver Ultra model surpasses GPT-4, a state-of-the-art generalist LLM, on various writing scenarios, demonstrating the advantage of training specialized LLMs for writing purposes.


Explaining a new state-of-the-art monocular depth estimation model: Depth Anything ✨ 🧶
Before we begin: Depth Anything is recently integrated to 🤗 transformers and you can use it with three lines of code! ✨
```python
from transformers import pipeline

pipe = pipeline(task="depth-estimation", model="LiheYoung/depth-anything-small-hf")
depth = pipe(image)["depth"]
```
We have also built an app for you to compare different depth estimation models 🐝 🌸  https://huggingface.co/spaces/merve/compare_depth_models
Check out Depth Anything in Web by @Xenova https://huggingface.co/spaces/Xenova/depth-anything-web

The model's success heavily depends on unlocking the use of unlabeled datasets, although initially the authors used self-training and failed.
What the authors have done:
➰ Train a teacher model on labelled dataset
➰ Guide the student using teacher and also use unlabelled datasets pseudolabelled by the teacher
However, this was the cause of the failure, as both architectures were similar, the outputs were the same.
So the authors have added a more difficult optimization target for student to learn additional knowledge on unlabeled images that went through color jittering, distortions, Gaussian blurring and spatial distortion, so it can learn more invariant representations from them.
The architecture consists of DINOv2 encoder to extract the features followed by DPT decoder. At first, they train the teacher model on labelled images, and then they jointly train the student model and add in the dataset pseudo-labelled by ViT-L.
Thanks to this, Depth Anything performs very well! I have also benchmarked the inference duration of the model against different models here. I also ran torch.compile benchmarks across them and got nice speed-ups 🚀 https://huggingface2.notion.site/DPT-Benchmarks-1e516b0ba193460e865c47b3a5681efb?pvs=4

osanseviero clem Xenova Ar4ikov tusk3d marooz


TIL: https://huggingface.co/datasets/EleutherAI/pile is on Wikipedia: https://en.wikipedia.org/wiki/The_Pile_(dataset)


🔍 Today's pick in Interpretability & Analysis of LMs: In-Context Language Learning: Architectures and Algorithms by E. Akyürek, B. Wang, Y. Kim, J. Andreas

This work methodically evaluates of in-context learning on formal languages across several model architectures, showing that Transformers outperform all other recurrent and convolutional models, including SSMs. These results are attributed to the presence of “n-gram heads” able to retrieve
the token following a context already seen in the current context window and copy it. This idea is further supported by a better ability of Transformer models to encode in-context n-gram frequencies for n>1, and a higher similarity of Transformer-based LM outputs with classical n-gram models trained on the same data. Finally, these insights are applied to the design static attention layers mimicking the behavior of n-gram head, which lead to lower perplexity despite the lower computational costs.

📄 Paper: https://huggingface.co/papers/2401.12973
💻 Code: https://github.com/berlino/seq_icl


Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation
  in Natural Language Generation (2302.09664)

Pretty novel idea on how to estimate *semantic* uncertainty. 🤔

Text generation tasks are challenging because a sentence can be written in multiple ways but still preserve its meaning.

For instance, "France's capital is Paris" means the same as "Paris is France's capital." 🇫🇷

In uncertainty quantification, we often look at token-level probabilities to quantify how "confident" an LLM is about its output. However, in this paper, the authors look at uncertainty at a meaning level.

Their motivation is that meanings are especially important for LLMs' trustworthiness; a system can be reliable even with many different ways to say the same thing, but answering with inconsistent meanings shows poor reliability.

To estimate semantic uncertainty, they introduce an algorithm for clustering sequences that mean the same thing, based on the principle that two sentences mean the same thing if we can infer one from the other. 🔄🤝

Then, they determine the likelihood of each meaning and estimate the semantic entropy by summing probabilities that share a meaning.

There's a lot more to it, but their results look quite nice when compared with non-semantic approaches.

Paper: https://huggingface.co/papers/2302.09664

osanseviero victor gsarti merve clem Citaman


The next version of Gradio will be significantly more efficient (as well as a bit faster) for anyone who uses Gradio's streaming features. Looking at you chatbot developers @oobabooga @pseudotensor :)

The major change that we're making is that when you stream data, Gradio used to send the entire payload at each token. This is generally the most robust way to ensure all the data is correctly transmitted. We've now switched to sending "diffs" --> so at each time step, we automatically compute the diff between the most recent updates and then only send the latest token (or whatever the diff may be). Coupled with the fact that we are now using SSE, which is a more robust communication protocol than WS (SSE will resend packets if there's any drops), we should have the best of both worlds: efficient *and* robust streaming.

Very cool stuff @aliabid94! PR: https://github.com/gradio-app/gradio/pull/7102

abbasm2 osanseviero yjernite oobabooga samusenps jeffboudier gsarti sam-kap satpalsr akhaliq alielfilali01 joaogante


"The Case for Co-Designing Model Architectures with Hardware"

This is a long overdue paper that we have started discussing back when training BLOOM-176.

Basically this paper tells you how to design your model's dimensions for an optimal training throughput.

Fantastic!

Yours truly contributed the SwiGLU section ;)

https://twitter.com/QuentinAnthon15/status/1752393989813375119

https://arxiv.org/abs/2401.14489

clem qanthony radames osanseviero abidlabs samusenps jeffboudier efederici merve muhtasham

clem osanseviero abidlabs merve muhtasham


So impressed with the speed and accuracy of https://huggingface.co/spaces/vikhyatk/moondream1 by
@vikhyatk (especially the last answer 😝😝😝).

Open multi-modal models have gone a long way!

Model: https://huggingface.co/vikhyatk/moondream1

vikhyatk victor osanseviero clem GloriaMK fffiloni akhaliq karankatke masonchambersiii abidlabs samusenps floattingabby gate369 jeffboudier ArthurZ combox snarktank michaelkleyn ivanfioravanti


Delighted to share a course I've learned a ton from about getting better outputs from LLMs

https://www.wandb.courses/courses/steering-language-models

We released it last Thursday (free) and at just 30 minutes of content total, its very information-dense with non-stop learnings covering important concepts around LLM validation, making your approach to LLM prompting more pythonic and quickly covers a basic RAG application at the end.

Would love to hear what ye think!

clem pierrci osanseviero altryne param-bharat merve mrdbourke kurianbenoy dillfrescott

clem osanseviero param-bharat merve dillfrescott


Quick build of the day: LCM Supa Fast Image Variation
—
We take the opportunity to combine moondream1 vision and LCM SDXL fast abilities to generate a variation from the subject of the image input.
All that thanks to gradio APIs 🤗

Try the space: https://huggingface.co/spaces/fffiloni/lcm-img-variations

samusenps victor clem osanseviero designfailure DamarJati merve


InternLM-XComposer2: Mastering Free-form Text-Image Composition and
  Comprehension in Vision-Language Large Model (2401.16420)

InternLM-XComposer2

Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model

paper page: https://huggingface.co/papers/2401.16420

Experimental results demonstrate the superiority of InternLM-XComposer2 based on InternLM2-7B in producing high-quality long-text multi-modal content and its exceptional vision-language understanding performance across various benchmarks, where it not only significantly outperforms existing multimodal models but also matches or even surpasses GPT-4V and Gemini Pro in certain assessments.


Black-Box Access is Insufficient for Rigorous AI Audits (2401.14446)

🔍 Today's pick in Interpretability & Analysis of LMs: Black-Box Access is Insufficient for Rigorous AI Audits by @stecas @carsonezell et al.

Audits conducted on AI systems can identify potential risks and ensure their compliance to safety requirements. Authors categorise audits based on the access to model-related resources (black, grey, white and out-of-the box) and highlight how levels of transparency on audited AI system enable broader and more effective auditing procedures. Technical, physical, and legal safeguards for performing audits are also introduced to ensure minimal security risks for audited companies. Authors conclude that transparency on the type of auditors’ access and methods is a pre-requisite to correctly interpret audit results, and white- and outside-the-box access allow for substantially more scrutiny than black-box access alone.

📄 Paper: https://huggingface.co/papers/2401.14446

🔍 Further readings:

📄Taxonomy of AI system access: https://bit.ly/struct-access
💻An API for transparent science on Black-box AI (NNsight): https://nnsight.net/about


EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models (2312.06281)

The most interesting LLM benchmark I've seen so far... reminder that there's lots of characterization of LLMs still yet to do.

https://huggingface.co/papers/2312.06281

bradwray wuyinansooymilk osanseviero clem doraemom0812


Fascinating paper by Rand shows that there is no statistically significant difference between using LLMs or regular internet to craft operational plans for bioweapons!

This is the first paper that actually studies the impact of AI on bioweapons from an operational perspective and looks at the big question: is AI any better than just using public data on the Internet?

As most of the data is most likely out there, an LLM would just be a more efficient tool to come up with the relevant information, but it seems that its impact is limited.

https://www.rand.org/pubs/research_reports/RRA2977-2.html

osanseviero samusenps clem Dlbk rreed-pha merve


SliceGPT: Compress Large Language Models by Deleting Rows and Columns (2401.15024)

SliceGPT: Compress Large Language Models by Deleting Rows and Columns

https://huggingface.co/papers/2401.15024

Large language models have become the cornerstone of natural language processing, but their use comes with substantial costs in terms of compute and memory resources. Sparsification provides a solution to alleviate these resource constraints, and recent works have shown that trained models can be sparsified post-hoc. Existing sparsification techniques face challenges as they need additional data structures and offer constrained speedup with current hardware. In this paper we present SliceGPT, a new post-training sparsification scheme which replaces each weight matrix with a smaller (dense) matrix, reducing the embedding dimension of the network. Through extensive experimentation, we show that SliceGPT can remove up to 25% of the model parameters (including embeddings) for LLAMA2-70B, OPT 66B and Phi-2 models while maintaining 99%, 99% and 90% zero-shot task performance of the dense model respectively. Our sliced models run on fewer GPUs and run faster without any additional code optimization: on 24GB consumer GPUs we reduce the total compute for inference on LLAMA2-70B to 64% of that of the dense model; on 40GB A100 GPUs we reduce it to 66%. We offer a new insight, computational invariance in transformer networks, which enables SliceGPT and we hope it will inspire and enable future avenues to reduce memory and computation demands for pre-trained models.

osanseviero vladbogo taufiqdp victor CHianogoi omaryshchenko AdinaY merve Locutusque Ji-Ha Saleh


With the Google announcement last week, I think we're now officially the only AI startup out there who has commercial collaborations with all the major cloud providers (AWS, GCP, Azure) and hardware providers (Nvidia, AMD, Intel, Qualcomm,...), making our vision of being the independent and agnostic platform for all AI builders truer than ever!

Let's go!

on1onmangoes osanseviero santiviquez mvaloatto abbasm2 donbale gsarti vladbogo MadElf1337 SivilTaram taufiqdp rreed-pha victor neovalle qnixsynapse andrewrreed vivekrp merve joaogante Bondstreet LucasThil maywell duraad hysts thibaultM


Confidence * may be * all you need.

A simple average of the log probabilities of the output tokens from an LLM might be all it takes to tell if the model is hallucinating.🫨

The idea is that if a model is not confident (low output token probabilities), the model may be inventing random stuff.

In these two papers:
1. https://aclanthology.org/2023.eacl-main.75/
2. https://arxiv.org/abs/2303.08896

The authors claim that this simple method is the best heuristic for detecting hallucinations. The beauty is that it only uses the generated token probabilities, so it can be implemented at inference time ⚡

clem leegao19 osanseviero victor s3nh abbasm2 gsarti rreed-pha stolsvik


TURNA: the biggest Turkish encoder-decoder model up-to-date, based on UL2 architecture, comes in 1.1B params 🐦 😍
The researchers also released models fine-tuned on various downstream tasks including text categorization, NER, summarization and more! 🤯 Great models @onurgu @gokceuludogan @yirmibesogluz @furkanakkurt1618 @uskudarli 👏
Fine-tuned models are in this collection 👉 https://huggingface.co/collections/boun-tabi-LMG/turna-ft-65b3f20aff5235e6cad07c1b
Pre-trained models are in this collection 👉 https://huggingface.co/collections/boun-tabi-LMG/turna-65ad340e5df673eec66e48c7

clefourrier mertyanik clem osanseviero gokceuludogan ridvanpal

osanseviero s3nh ktdilankaya ismaildlml ridvanpal


Sentence Transformers v2.3.0 has been released! It includes several bug fixes, enhanced model loading including custom models & no more unnecessary file downloads, improved performance, a powerful loss function, and much more!

Details:
⬆ Uploading Models to the Hub with `save_to_hub`.
⬇ Downloading Models from the Hub now downloads only necessary files.
⚙ Custom Models (such as https://huggingface.co/jinaai/jina-embeddings-v2-base-de) can now be loaded with `trust_remote_code=True`.
🔍 Models can now be loaded at specific revisions (e.g. commit hashes or git branches).
🖥 Various device fixes; models will now always operate on the device that you specify.
📉 A new "Cached" variant of the powerful Multiple Negatives Ranking Loss allows common hardware to reach performance previously only accessible on multi-gpu clusters.
🐎 Computation time of Community Detection was decreased significantly (7x speedup at 500k sentences :exploding_head:)
🪶 Removed the now unnecessary "torchvision" dependency for a smaller installation.

Check out the full changelog here: https://github.com/UKPLab/sentence-transformers/releases/tag/v2.3.0

I'll be working on much more changes in the near future, so expect more exciting updates. If you encounter any issues, or have any questions or feature requests, don't hesitate to open an issue on the repository: https://github.com/UKPLab/sentence-transformers/issues

osanseviero clem victor chkla santiviquez lbourdois djsull taufiqdp shm141 nickprock hysts


🔥 New LLM leaderboard on the hub: an LLM Hallucination Leaderboard!

Led by @pminervini , it evaluates the propensity of models to *hallucinate*, either on factuality (= say false things) or faithfulness (= ignore user instructions). This is becoming an increasingly important avenue of research, as more and more people are starting to rely on LLMs to find and search for information!
It contains 14 datasets, grouped over 7 concepts, to try to get a better overall view of when LLMs output wrong content.
https://huggingface.co/spaces/hallucinations-leaderboard/leaderboard?logs=container

Their introductory blog post also contains an in depth analysis of which LLMs get what wrong, which is super interesting: https://huggingface.co/blog/leaderboards-on-the-hub-hallucinations

Congrats to the team! 🚀


Current ranking of pre-trained (nonchat) open access LLMs according to the leaderboard. 1-4 are from China-based groups. Does training models with Chinese somehow lead to better metrics? 🤔 WDYT?

1. https://huggingface.co/Qwen/Qwen-72B
2. https://huggingface.co/01-ai/Yi-34B/tree/main
3. https://huggingface.co/internlm/internlm2-20b
4. https://huggingface.co/deepseek-ai/deepseek-llm-67b-base
5. https://huggingface.co/mistralai/Mixtral-8x7B-v0.1

samusenps GregorG abidlabs julien-c admarcosai mishig clefourrier AdinaY shm141 Xhehab

KnutJaegersberg julien-c mhammadkhan aloobun AdinaY zhniu


🔎 DPO hyperparameter search update!

In my previous post (https://huggingface.co/posts/BramVanroy/633544255876795), I indicated how despite high reward accuracies and low losses, my model would sometimes just output repeating random tokens (`/*****/`). There were some useful brainstorms in that thread. I think the dataset is relatively easy for the model, leading it to quickly overfit when the beta is very small, which allows the model to step away further from its initially outputs.

So, I ran a hyperparameter search for learning rate (1e-7 v 5e-7), batch size (32, 64, 96, 128) and most importantly, beta (0.01, 0.1, 0.2, 0.5). You can have a look at the results for yourself here: https://wandb.ai/bramvanroy/dpo-geitje-ultra-hyperparams

Interpreting the result, I'd think that the beta=0.5 is the better choice for this dataset. Reasons:

- markedly higher rewards margins compared to all other betas
- better balance between positive chosen and negative rejected rewards
- log probabilities are not as superbly low as for beta=0.01, which seems too low for this dataset

Of course, that is just purely looking at numbers without running any benchmarks. However, I am hesitant to evaluate all the models on benchmarks and, therefore, literally optimising my hyperparameters on a test set (which is very bad!). So I will just play with some of the most promising models and see which one feels "best" qualitatively.

If you have other insights, thoughts, or opinions, let me know!

osanseviero tomaarsen kashif edbeeching davanstrien philschmid dingo-actual Rijgersberg dvilasuero gabrielmbmb abidlabs julien-c HumanFace Cherran


LLMCheckup: Conversational Examination of Large Language Models via
  Interpretability Tools (2401.12576)

🔍 Today's pick in Interpretability & Analysis of LMs: LLMCheckup: Conversational Examination of Large Language Models via Interpretability Tools by @qiaw99 @tanikina @nfel et al.

Authors introduce LLMCheckup, a conversational interface connecting an LLM to several interpretability tools (feature attribution methods, similarity, counterfactual/rationale generation) allowing users to inquire about LLM predictions using natural language. The interface consolidates several interpretability methods in a unified chat interface, simplifying future investigations into natural language explanations.

📄 Paper: https://huggingface.co/papers/2401.12576
💻 Code: https://github.com/DFKI-NLP/LLMCheckup
🎥 Demo video: https://www.youtube.com/watch?v=ZwN8ZQSXoOU

samusenps osanseviero qiaw99 nfel Citaman compressionsavant Saw


Presenting: SimpleMath

Recently we uploaded on the hub our LATEST and most powerful version of SimpleMath SFT dataset.
Today we are happy to present SimpleMath DPO Pairs, improving further mathematical capabilities on LLM's.

Our first results shows clear improvements on GSM8k, MATHQA, ARC, TQA, MMLU and BBH. Feel free to experiment and generate your own dataset, as we also provide the code to generate them synthetically.

https://huggingface.co/datasets/fblgit/simple-math
https://huggingface.co/datasets/fblgit/simple-math-DPO
https://huggingface.co/fblgit/UNA-34BeagleSimpleMath-32K-v1

samusenps XaevrM sehyun66 fblgit osanseviero victor jtatman


A 1b dense causal language model begins to "saturate" in terms of accuracy around 5 epochs on 1.2T tokens.


Are you working with users and would like to let them edit a text containing highlights? Check out my new Gradio component `HighlightedTextbox`! 🤗

The component can be installed with `pip install gradio_highlightedtextbox` and used seamlessly with other native components in Gradio interfaces. It supports texts with multiple tags and provides some reasonable UX behaviors (tags disappear when an edit is performed inside them). It should be great for UIs to study user editing of annotated LM generations/translations!

Demo: https://huggingface.co/spaces/gsarti/gradio_highlightedtextbox

osanseviero pngwn abidlabs samusenps hutusi victor freddyaboulton

abidlabs osanseviero hutusi julien-c freddyaboulton


👋hi there folks !

check out this Chest X-Ray model from AIMIStanford : https://huggingface.co/spaces/Tonic/CheXRay

thanks to @lunarflu for kicking me a bit to get the examples in there !

would be great to get even more examples and even more downstream functions , so contributions are very welcome, or if you have a dataset source, please do share it in the discussions !

samusenps osanseviero KaiKimera peaceAsh victor clem lunarflu matlok


Exploring the Reasoning Abilities of Multimodal Large Language Models
  (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning (2401.06805)

COCO is "ALL'' You Need for Visual Instruction Fine-tuning (2401.08968)

Thrilled to share some of our recent work in the field of Multimodal Large Language Models (MLLMs).

1⃣ A Survey on Multimodal Reasoning 📚
Are you curious about the reasoning abilities of MLLMs? In our latest survey, we delve into the world of multimodal reasoning. We comprehensively review existing evaluation protocols, categorize the frontiers of MLLMs, explore recent trends in their applications for reasoning-intensive tasks, and discuss current practices and future directions. For an in-depth exploration, check out our paper: https://huggingface.co/papers/2401.06805

2⃣ Advancing Flamingo with InfiMM 🔥
Building upon the foundation of Flamingo, we introduce the InfiMM model series. InfiMM is a reproduction of Flamingo, enhanced with stronger Large Language Models (LLMs) such as LLaMA2-13B, Vicuna-13B, and Zephyr7B. We've meticulously filtered pre-training data and fine-tuned instructions, resulting in superior performance on recent benchmarks like MMMU, InfiMM-Eval, MM-Vet, and more. Explore the power of InfiMM on Huggingface: https://huggingface.co/Infi-MM/infimm-zephyr

3⃣ Exploring Multimodal Instruction Fine-tuning 🖼
Visual Instruction Fine-tuning (IFT) is crucial for aligning MLLMs' output with user intentions. Our research identified challenges with models trained on the LLaVA-mix-665k dataset, particularly in multi-round dialog settings. To address this, we've created a new IFT dataset with high-quality, diverse instruction annotations and images sourced exclusively from the COCO dataset. Our experiments demonstrate that when fine-tuned with this dataset, MLLMs excel in open-ended evaluation benchmarks for both single-round and multi-round dialog settings. Dive into the details in our paper: https://huggingface.co/papers/2401.08968

Stay tuned for more exciting developments.
Special thanks to all our collaborators: @Ye27 @wwyssh @Yongfei @Yi-Qi638 @xudonglin @KhalilMrini @lllliuhhhhggg @Borise @Hongxia

Ye27 wwyssh samusenps lllliuhhhhggg osanseviero victor xudonglin xiaotianhan Tonic yarth69 clem lunarflu

abacaj Tonic clem lunarflu xiaotianhan Yi-Qi638


New Book: Building Disruptive AI & LLM Technology from Scratch https://mltblog.com/404F1BZ

This book features new advances in game-changing AI and LLM technologies built by GenAItechLab.com. Written in simple English, it is best suited for engineers, developers, data scientists, analysts, consultants and anyone with an analytic background interested in starting a career in AI. The emphasis is on scalable enterprise solutions, easy to implement, yet outperforming vendors both in term of speed and quality, by several orders of magnitude.

Each topic comes with GitHub links, full Python code, datasets, illustrations, and real-life case studies, including from Fortune 100 company. Some of the material is presented as enterprise projects with solution, to help you build robust applications and boost your career. You don’t need expensive GPU and cloud bandwidth to implement them: a standard laptop works.

➡ Part 1: Hallucination-Free LLM with Real-Time Fine-Tuning

➡ Part 2: Outperforming Neural Nets and Classic AI

➡ Part 3: Innovations in Statistical AI

About the author

Vincent Granville is a pioneering GenAI scientist and machine learning expert, co-founder of Data Science Central (acquired by a publicly traded company in 2020), Chief AI Scientist at ML Techniques and GenAI Techlab, former VC-funded executive, author (Elsevier) and patent owner — one related to LLM. Vincent’s past corporate experience includes Visa, Wells Fargo, eBay, NBC, Microsoft, and CNET.

➡ See content and get your copy, at https://mltblog.com/404F1BZ


Don't use an LLM when you can use a much cheaper model.

The problem is that no one tells you how to actually do it.

Just picking a pre-trained model (e.g., BERT) and throwing it at your problem won't work!

If you want a small model to perform well on your problem, you need to fine-tune it.

And to fine-tune it, you need data.

The good news is that you don't need a lot of data but instead high-quality data for your specific problem.

In the latest livestream, I showed you guys how to get started with Argilla on the Hub! Hope to see you at the next one.

https://www.youtube.com/watch?v=BEe7shiG3rY


AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents (2410.09024)

🚨 New Agent Benchmark 🚨
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents

https://huggingface.co/datasets/ai-safety-institute/AgentHarm

Collaboration between UK AI Safety Institute and Gray Swan AI to create a dataset for measuring harmfulness of LLM agents.

The benchmark contains both harmful and benign sets of 11 categories with varied difficulty levels and detailed evaluation, not only testing success rate but also tool level accuracy.

We provide refusal and accuracy metrics across a wide range of models in both no attack and prompt attack scenarios.

https://huggingface.co/papers/2410.09024


New York Times to Perplexity: Stop Using Our Stuff

The publisher has sent generative-AI startup Perplexity a “cease and desist” notice demanding that the firm stop accessing and using its content, according to a copy of the letter reviewed by The Wall Street Journal.

Perplexity CEO Aravind Srinivas said in an interview that Perplexity isn’t ignoring the Times’s efforts to block crawling of its site. He said the company plans on responding to the legal notice by the Times’s deadline of Oct. 30.

“We are very much interested in working with every single publisher, including the New York Times,” Srinivas said. “We have no interest in being anyone’s antagonist here.”

https://www.wsj.com/business/media/new-york-times-to-bezos-backed-ai-startup-stop-using-our-stuff-20faf2eb?mod=rss_Technology


I just published the first article in a pair. I could make it a longer tailed series, in case you liked em. This one dives into self-hosting Metaflow without needing S3, illustrated with a version tailored for Google Colab.
find it @ https://huggingface.co/blog/Aurelien-Morgan/stateful-metaflow-on-colab


👋 We're excited to share our project on mobile face detection using MediaPipe and ZETIC.MLange.

🔑 Key highlights:
1. Introduction to Mediapipe face detection model
2. Developing on-device AI applications with ZETIC.MLange
3. Guide to creating object detection apps utilizing Mobile NPUs

📱 Learn how to build a high-performance face detection app that operates entirely on-device, no cloud required! We explore:
- Real-time face analysis techniques
- Enhanced security measures
- Privacy protection strategies

🎥 Check out our demo video showcasing real-time face detection:
Face detection Demo: https://youtu.be/GXtJKk7MdjQ?si=41UFKL8IBwx5nlxs

📚 Full Tutorial
For a step-by-step guide and in-depth discussion, read our full blog post:
https://zetic.ai/blog/implementing-face-detection-on-device-ai-with-zetic-mlange

We'd love to hear your thoughts, experiences, or questions in the comments below.


Good folks from Universitat Politècnica de Catalunya, University of Groningen, and Meta have released "A Primer on the Inner Workings of Transformer-based Language Models."

They don't make survey papers like they used to, but this is an exciting new survey on Transformer LM interpretability!

This comprehensive survey provides a technical deep dive into:

• Transformer architecture components (attention, FFN, residual stream)
• Methods for localizing model behavior:
  - Input attribution (gradient & perturbation-based)
  - Component importance (logit attribution, causal interventions)
• Information decoding techniques:
  - Probing, linear feature analysis
  - Sparse autoencoders for disentangling features
• Key insights on model internals:
  - Attention mechanisms (induction heads, copy suppression)
  - FFN neuron behaviors
  - Residual stream properties
  - Multi-component emergent behaviors

The paper offers a unified notation and connects insights across different areas of interpretability research. It's a must-read for anyone working on understanding large language models!

Some fascinating technical highlights:
- Detailed breakdowns of attention head circuits (e.g., IOI task)
- Analysis of factual recall mechanisms
- Overview of polysemanticity and superposition
- Discussion of grokking as circuit emergence

What interpretability insights do you find most intriguing?


Transformers are not all we need, that is being proven repeatedly now as more alternative frameworks emerge. Another such framework is Kolmogorov Arnold Network based Transformers. I break down exactly how these differ from Perceptron based Transformers and give you the link to my Colab where I create a model based on the research paper that absolutely destroys a standard Transformers based model. Check out the video here: https://www.youtube.com/watch?v=Sw0euxNZCc4


To demonstrate that it was possible, I performed a "trapezoid" gradient merge of a Llama 3 8B model onto Llama 3.1 8B Instruct, favoring the L3.1 model at the ends in order to preserve coherence and limiting the influence of the L3 model to at most 0.1 weight. Tested to 16k context length.
https://huggingface.co/grimjim/Llama-Nephilim-Metamorphosis-v2-8B


🎙 Introducing LiveATC Recordings (Partial 2024-08-26) Dataset - https://huggingface.co/datasets/nyuuzyou/liveatc

Dataset highlights:

- 21,172 air traffic control audio recordings from LiveATC.net for August 26, 2024
- Multilingual content, primarily in English with potential for other languages
- Each entry includes: audio file, ICAO airport code, facility type, date, and time
- Contains original MP3 files stored in .tar.zst archives, organized by ICAO airport code
- Data covers various airports and ATC facilities worldwide
- Subject to LiveATC.net's Terms of Use for personal, non-commercial use only

The dataset can be used for audio classification, automatic speech recognition, and analysis of air traffic control communications. The inclusion of recordings from multiple airports allows for comparative analysis across different locations and facility types.


🔥 New LLM leaderboard on the hub: an LLM Safety Leaderboard!

It evaluates LLM safety, such as bias and toxicity, PII, and robustness, and is powered by DecodingTrust (outstanding paper at Neurips!) 🚀
https://huggingface.co/spaces/AI-Secure/llm-trustworthy-leaderboard

It's great to see such initiatives emerge, trying to understand the risks and biases of LLMs, and I'm hoping other tools will follow. It should be interesting for the community of model builders (whether or not they want uncensored models ^^).

Detailed intro blog: https://huggingface.co/blog/leaderboards-on-the-hub-decodingtrust.

Congrats to the AI Secure team!

clem samusenps monsoon-nlp osanseviero neovalle mc0ps gsarti admarcosai sbrandeis


Gradio 4.16 introduces a new flow: you can hide/show Tabs or make them interactive/non-interactive.

Really nice for multi-step machine learning ademos ⚡

clem samusenps osanseviero gsarti Shivansh000 jiaowobaba02 freddyaboulton


Hello everyone,
This is my first post! I have also decided to release a dataset that I have been keeping private for a while now. I’ve kept it private because I’m not sure if it is actually good or not. I would greatly appreciate it if someone could fine-tune some larger models and evaluate the dataset. Named Hercules-v1.0, it is a turbo-charged version of teknium’s openhermes generated by augmenting its data sources. Learn more in the dataset card: https://huggingface.co/datasets/Locutusque/hercules-v1.0

osanseviero samusenps Locutusque abidlabs clem ujjal13 admarcosai

clem Felladrin afrideva Locutusque maxamed sbrandeis


In 2024, we're expanding from open weights to open EVERYTHING (datasets, training scripts,...).

Excited to see this dataset release in French by @Pclanglais  @carbonbasedLLM @anastasiastasenko:
https://huggingface.co/datasets/PleIAs/French-PD-Newspapers

"To give you an idea of the size, the full French Wikipedia is about 2 billon words. This is 40 times larger."

Maani Pclanglais afrideva tomaarsen osanseviero samusenps glenn-parham mlabonne ucalyptus abidlabs Sylvestre JaiSurya anastasiastasenko mvaloatto sparkycollier linoyts MexIvanov AdinaY brishtiteveja


Diffuse to Choose: Enriching Image Conditioned Inpainting in Latent
  Diffusion Models for Virtual Try-All (2401.13795)

Diffuse to Choose

Enriching Image Conditioned Inpainting in Latent Diffusion Models for Virtual Try-All

paper page: https://huggingface.co/papers/2401.13795

Diffuse to Choose (DTC) allows users to virtually place any e-commerce item in any setting, ensuring detailed, semantically coherent blending with realistic lighting and shadows.

blanchon clem mvaloatto Danyray101 osanseviero samusenps ucalyptus Shivansh000


The Calibration Gap between Model and Human Confidence in Large Language
  Models (2401.13835)

🔍 Today's pick in Interpretability & Analysis of LMs: The Calibration Gap between Model and Human Confidence in Large Language Models by @coolprof H. Tejeda A. Kumar @Cbelem @skarny et al.

This work involves an experimental study to assess human confidence in LLM responses to multiple-choice MMLU questions based on explanations the LLM provides together with the selected answer. The authors experiment with altering the model prompt to reflect the actual prediction confidence in models’ explanations, showing improved calibration for users’ assessment of LLM’s reliability and a better ability to discriminate between correct and incorrect answers. These results suggest the importance of further research on the impact of automatic explanations on users’ perception.

📄 Paper: https://huggingface.co/papers/2401.13835

osanseviero clem victor clefourrier afrideva skarny samusenps aakritikumar adhisetiawan sbrandeis


If you have to choose one small base language model <=3B for ChatML Code Assistant (SFT+DPO) to validate the approach on the dataset and tune hyperparams, so later retrain with a larger base model like Mistral/Mixtral, what model would you pick?
🧵

afrideva Noahosmonth joshpxyne sbrandeis


Hi everyone,
For my first post, I'm announcing a big release (in multiple ways): probably the largest open corpus in French to date, with 85 billion words in the public domain.
The dataset has been prepared in collaboration with Benoît de Courson and Benjamin Azoulay from Gallicagram (https://shiny.ens-paris-saclay.fr/app/gallicagram). Gallicagram is a major cultural analytics project in French, the open and better version of ngram viewer for large scale search of word and ngram occurrences.
The corpus is made of two different dataset for monographs (16B words) https://huggingface.co/datasets/PleIAs/French-PD-Newspapers and newspapers/periodicals (69B) https://huggingface.co/datasets/PleIAs/French-PD-Newspapers Along with the full text it also includes core provenance metadata.
Beyond research in digital humanities, the corpus can also be used to training open and reproducible LLMs. Being in the public domain means it can be released everywhere in any shape without restrictions.
The corpus is not perfect: digitization of cultural heritage is challenging and, especially for newspapers, we tackle with layout issues and a significant rate of optical character recognition mistake. Our conviction is that releasing corpus as a commons is the best way to improve on this. Sharing is caring.

anastasiastasenko davanstrien julien-c osanseviero clem victor Tonic Pclanglais blanchon bgowan philschmid kramp Wauplin clefourrier afrideva samusenps KaiKimera kheopss yjernite sbrandeis

davanstrien julien-c osanseviero clem kramp vicgalle Tonic blanchon philschmid damerajee danielus

neovalle julien-c osanseviero clem Tonic samusenps damerajee


My TED talk is finally live!! I proposed the recipe for the "Foundation Agent": a single model that learns how to act in different worlds. LLM scales across lots and lots of texts. Foundation Agent scales across lots of lots of realities. If it is able to master 10,000 diverse simulated realities, it may well generalize to our physical world, which is simply the 10,001st reality.

Why do we want a single Foundation Agent instead of many smaller models? I'd like to quote the idea from my friend Prof. Yuke Zhu's CoRL keynote talk. If we trace each AI field's evolution, we'd find this pattern:

Specialist -> Generalist -> Specializing Generalist

And the "specialized generalist" is often way more powerful than the original specialist. Just like distilled versions of LlaMA are way better than custom-built NLP systems 5 years ago.

TED talks do not have teleprompters!! All I have is a "confidence monitor" at my foot, showing only the current slide and timer. That means I need to memorize the whole speech. It sounds intimidating at first, but turns out to be the best way to connect with the audience and deliver the ideas right to heart.

Happy to share my slides with all of you! https://drive.google.com/file/d/1NSY6MxMu3OPQ4U6hx0OxPq7EQB5XTcAG/view?usp=sharing

The video is only 10 min. I promise it's well worth your time!
https://www.ted.com/talks/jim_fan_the_next_grand_challenge_for_ai

truong-xuan-linh s3nh gsarti samusenps osanseviero clem victor julien-c akhaliq juancopi81 afrideva frankdata

samusenps osanseviero clem julien-c akhaliq sbrandeis


Do you have a hidden massive storage leak thanks to HF hub models and datasets revisions adding up and not getting automatically deleted?

Here is how to delete all old revisions and only keeping `main` in a few quick steps and no tedious manual editing.

In terminal A:
```
$ pip install huggingface_hub["cli"] -U
$ huggingface-cli delete-cache --disable-tui
File to edit: /tmp/tmpundr7lky.txt
0 revisions selected counting for 0.0. Continue ? (y/N)
```
Do not answer the prompt and proceed with my instructions.

(note your tmp file will have a different path, so adjust it below)

In terminal B:
```
$ cp /tmp/tmpedbz00ox.txt cache.txt
$ perl -pi -e 's|^#(.*detached.*)|$1|' cache.txt
$ cat cache.txt >>  /tmp/tmpundr7lky.txt
```
The perl one-liner uncommented out all lines that had `(detached)` in it - so can be wiped out. And then we pasted it back into the tmp file `huggingface-cli` expects to be edited.

Now go back to terminal A and hit: N, Y, Y, so it looks like:

```
0 revisions selected counting for 0.0. Continue ? (y/N) n
89 revisions selected counting for 211.7G. Continue ? (y/N) y
89 revisions selected counting for 211.7G. Confirm deletion ? (Y/n) y
```
Done.

If you messed up with the prompt answering you still have `cache.txt` file which you can feed again to the new tmp file it'll create when you run `huggingface-cli delete-cache --disable-tui` again.

For more details and additional techniques please see https://github.com/stas00/ml-engineering/tree/master/storage#huggingface-hub-caches

vicgalle cahlen osanseviero not-lain samusenps clem victor Wauplin afrideva sethuiyer sbrandeis


New in Gradio 4.16.0 - Galleries as Input 🖼

Now your users can upload multiple images as input to your AI application and view them in a slick gallery!

Attached is a demo of how this new feature can be used in a photomaker-type application: https://huggingface.co/spaces/TencentARC/PhotoMaker

Shout out @abidlabs and @akhaliq who proposed this feature after seeing some of the workarounds gradio developers were using in the wild to upload multiple images.

The gradio team works hard to stay up to date with the latest trends in AI! If there's something missing from the library, file an issue on github! https://github.com/gradio-app/gradio/issues

kramp ysharma sbrandeis samusenps afrideva baxin mrfakename Sattineez toyxyz


✨ Excited to release `gradio` 4.16. New features include:

🐻❄ Native support for Polars Dataframe
🖼 Gallery component can be used as an input
⚡ Much faster streaming for low-latency chatbots
📄 Auto generated docs for custom components

... and much more! This is HUGE release, so check out everything else in our changelog:  https://github.com/gradio-app/gradio/blob/main/CHANGELOG.md

clem sbarman25 osanseviero gsarti samusenps ysharma freddyaboulton


Google + Hugging Face + Open-Source AI = 🔥🔥🔥

https://huggingface.co/blog/gcp-partnership

https://finance.yahoo.com/video/google-hugging-face-alliance-spur-173016882.html

https://www.theverge.com/2024/1/25/24050445/google-cloud-hugging-face-ai-developer-access

https://www.bloomberg.com/news/articles/2024-01-25/google-to-team-up-with-startup-hugging-face-to-host-ai-software

https://www.reuters.com/technology/google-cloud-partners-with-hugging-face-attract-ai-developers-2024-01-25/

EmilyWitko osanseviero vladbogo neovalle FranzNo mrfakename radames akhaliq not-lain afrideva Sylvestre

osanseviero Paul-HF vicgalle akhaliq koisose stas not-lain taufiqdp sbrandeis

mrfakename akhaliq not-lain kramp mvaloatto


We just released the first Text2SQL model for DuckDB 🦆🧠
You can try it out directly here :
https://huggingface.co/spaces/motherduckdb/DuckDB-NSQL-7B

osanseviero samusenps lunarflu clem julien-c not-lain lhoestq Johnnycadelover lysandre bgowan afrideva meetrais jarvisx17 rreed-pha cnmoro severo notsahil sbrandeis

osanseviero lunarflu clem julien-c lhoestq koisose lysandre kramp mrfakename

osanseviero lunarflu julien-c lysandre Tonic afrideva rreed-pha


hey there folks , work in progress, but basically celebrating the release of `whisperspeech`

just :

```bash
pip install whisperspeech
```
to get started and check out my demo to do multilingual text to speech including making voice prints using `whisperspeech`reverse engineering of whisper here : https://huggingface.co/spaces/Tonic/whisperspeech
and the model card here : https://huggingface.co/collabora/whisperspeech

i met collabora on LAION check out LAION here :
https://huggingface.co/laion

lunarflu samusenps clem Johnnycadelover cahlen afrideva emilios rreed

julien-c osanseviero lunarflu clem rreed


I've just published the 23rd edition of the satellite-image-deep-learning newsletter to 8,188 subscribers

This edition: METEOR, Seeing the roads through the trees 🌴, A New Learning Paradigm for Foundation Model-based Remote Sensing Change Detection & Globe230k dataset

https://www.satellite-image-deep-learning.com/p/new-discoveries-23

andysalerno Tonic osanseviero samusenps clem abidlabs sbrandeis

Ch-Usman marinaretik Tonic osanseviero abidlabs


Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic
  Image Restoration In the Wild (2401.13627)

Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild

paper page: https://huggingface.co/papers/2401.13627

introduce SUPIR (Scaling-UP Image Restoration), a groundbreaking image restoration method that harnesses generative prior and the power of model scaling up. Leveraging multi-modal techniques and advanced generative prior, SUPIR marks a significant advance in intelligent and realistic image restoration. As a pivotal catalyst within SUPIR, model scaling dramatically enhances its capabilities and demonstrates new potential for image restoration. We collect a dataset comprising 20 million high-resolution, high-quality images for model training, each enriched with descriptive text annotations. SUPIR provides the capability to restore images guided by textual prompts, broadening its application scope and potential. Moreover, we introduce negative-quality prompts to further improve perceptual quality. We also develop a restoration-guided sampling method to suppress the fidelity issue encountered in generative-based restoration. Experiments demonstrate SUPIR's exceptional restoration effects and its novel capacity to manipulate restoration through textual prompts.


Model Editing Can Hurt General Abilities of Large Language Models (2401.04700)

🔍 Today's pick in Interpretability & Analysis of LMs: Model Editing Can Hurt General Abilities of Large Language Models by J.C. Gu et al.

This work raises concerns that gains in factual knowledge after model editing can result in a significant degradation of the general abilities of LLMs. The authors evaluate 4 popular editing methods on 2 LLMs across eight representative tasks, showing model editing does substantially hurt model general abilities. A suggestion is made to prioritize improvements in LLMs' robustness, developing more precise editing methods, and better evaluation benchmarks.

📄 Paper: https://huggingface.co/papers/2401.04700
💻 Code: https://github.com/JasonForJoy/Model-Editing-Hurt

merve vicgalle osanseviero Dlbk clem samusenps lunarflu


Can you merge models of different sizes? ⚗

Well, yes, if the models are somewhat compatible. Here is an experiment I did. I wanted to merge two of the best performing models: https://huggingface.co/mlabonne/NeuralBeagle14-7B and https://huggingface.co/jeonsworld/CarbonVillain-en-10.7B-v4

Here is my recipe:
1. Expand the layers of NeuralBeagle to 10.7B ala frankenmerge.
2. DPO-tune the previous model with a high-quality preference dataset, https://huggingface.co/datasets/argilla/distilabel-intel-orca-dpo-pairs
3. Merge the previous model with CarbonVillain (needs —allow-crimes in mergekit! 🔪)

And here is the resulting model, CarbonBeagle-11B, which ranked top in the leaderboard for its size class:
https://huggingface.co/vicgalle/CarbonBeagle-11B

vicgalle victor samusenps NeuralNovel itsn78 merve Dlbk clem antiven0m lunarflu Svngoku Makya mlabonne tryumanshow

osanseviero itsn78 merve clem cbensimon lunarflu luqmanxyz vicgalle Makya roinaveiro


Lumiere: A Space-Time Diffusion Model for Video Generation (2401.12945)

LUMIERE

A Space-Time Diffusion Model for Video Generation

paper page: https://huggingface.co/papers/2401.12945

Demonstrate state-of-the-art text-to-video generation results, and show that our design easily facilitates a wide range of content creation tasks and video editing applications, including image-to-video, video inpainting, and stylized generation.

osanseviero victor s3nh taufiqdp samusenps podevi merve clem blanchon

osanseviero wssb mvaloatto merve clem tengomucho


GPU Poor POV: Willingness of Customization

I love to use libraries in which you can customize a lot of things. Chromadb is my choice of db if it comes to store embeddings. Te cool feature is that you can define your own embeddings function which can be called on every chromadb collection initialisation or creation. It is useful because sometimes we want to use  different prompts, different models, and it can be easily written as inheritence from EmbeddingFunction class.

Edit:

My CustomEmbeddingFunction can be found here:
https://gist.github.com/s3nh/cfbbf43f5e9e3cfe8c3e4e2f0d550b80

and you can use it by initializing or calling the chroma collection.

```python

import chromadb
from your_custom_fn import CustomEmbeddingFunction
class ChromaStorage:
    def __init__(self, config):
        self.config = config
        self.client = self.init_client()
        self.embedding_function = CustomEmbeddingFunction()

    def check_config(self):
        assert os.path.exists(self.config.path), ValueError('Provided path does not exists!!')

    def init_client(self):
        return chromadb.PersistentClient(path = self.config.path,)

    def init_collection(self, name: str):
        return self.client.get_or_create_collection(name = name, embedding_function = self.embedding_function)
```

victor julien-c samusenps NeuralNovel krinal clem s3nh lunarflu KnutJaegersberg tawfikgh Ji-Ha

osanseviero victor julien-c NeuralNovel clem


👋 Hi there folks,

I launched my first competition !

Goal : Use AI to beat the Math Olympics within the set time

Basically we're looking for adventurous teams and individuals to make a common submission to the AI Math Olympics by the MLCommons.

Althought the ultimately there can only be one winner and there must always be a winner, the ultimate goal is to get together for a common solution.

check it out here :
https://huggingface.co/spaces/Tonic1/mathathon

s3nh samusenps osanseviero boinc victor Solshine clem lunarflu Ji-Ha


🏅 New top model on the GAIA benchmark!

Called FRIDAY, it's a mysterious new autonomous agent, which got quite good performances on both the public validation set *and* the private test set.
It notably passed 10 points for the val and 5 points for the test set on our hardest questions (level 3): they require to take arbitrarily long sequences of actions, use any number of tools, and access the world in genera! ✨

The GAIA benchmark evaluates next-generation LLMs (LLMs with augmented capabilities due to added tooling, efficient prompting, access to search, etc) and was co authored by @gregmialz @ThomasNLG @ylecun @thomwolf and myself: https://huggingface.co/spaces/gaia-benchmark/leaderboard

gregmialz osanseviero Tonic m-ric clem julien-c kahris

osanseviero Tonic sharad mariagrandury clem diwank m-ric


From Understanding to Utilization: A Survey on Explainability for Large
  Language Models (2401.12874)

🔍 Today's pick in Interpretability & Analysis of LMs: From Understanding to Utilization: A Survey on Explainability for Large Language Models by H. Luo and L. Specia

This survey summarizes recent works in interpretability research, focusing mainly on pre-trained Transformer-based LMs. The authors categorize current approaches as either local or global and discuss popular applications of LM interpretability, such as model editing, enhancing model performance, and controlling LM generation.

📄 Paper: https://huggingface.co/papers/2401.12874

osanseviero cihan30 samusenps aigclab clem lunarflu


Making gradio's auto-reload leaner and more robust 💪

Auto-reload is one of the coolest (and maybe underrated) features of gradio in my opinion. It automatically detects changes in your python app file and hot swaps your backend and frontend without restarting the server. I may be biased, but that's a way nicer experience than most other reload mode features out there.

This is all possible due to novel use of python's built in importlib module. Since this is a non-standard use of the module, our users have encountered some rough edges in the wild.

I'm happy to announce two improvements that will be out in the next release of gradio. They were both made possible by collaboration in the open source community.

1. First, gradio's reload mode now works with the `python-dotenv` library. Getting to the bottom of this one took a lot of sleuthing by our users and the fix got merged and released into the `python-dotenv` package last night! Thanks to @theskumar, maintainer of python-dotenv, for all the help.
2. Second, gradio's reload mode now consumes an order of magnitude less CPU. Depending on how many files are in your source directory, you may see between 2x to 10x less CPU utilization. Thanks to @velaia for filing the issue and @abidlabs for the review!

More improvements to reload mode are planned. Stay tuned!

clem abidlabs osanseviero victor vladbogo gsarti samusenps radames chansung akashicmarga mathiasn1 Dlbk Wauplin kramp clefourrier velaia hysts

abidlabs osanseviero clem radames chansung ashwinlimaye


🕵 Looking for DPO experts!

I have a dataset that is gpt-4-turbo as chosen and a lower performing model as rejected. The objective should therefore be fairly easy because the two are easy to discern. As a consequence, the model achieves very low losses (0.021 train; 0.013 validation) and high reward accuracies (0.995). **However**, when using the model in practice, it often detoriates after the first one or two tokens and continuously outputs sequences of `/*****/`. So despite the good performance on the DPO objective and strong scores on the validation set (no overfitting), something seems to go wrong. Perhaps the outputs are too different and the task is too easy, in which case DPO is not useful. But why then would the model start hallucinating and repeating the same token over and over again?

Any thoughts? Any suggestions to get around this? All discussions are welcome!

mrfakename davanstrien clem osanseviero lewtun Dlbk lilsyoss gsarti julien-c clefourrier lunarflu larryvrh


✅New paper to ensure valid LLM output with SOTA LLMs like GPT4 by mixing it with OSS LLMs

Paper: arxiv.org/abs/2401.09967

Great paper showing how strong proprietary AI like #GPT4 can be paired with #OSS LLM to ensure LLM output validity, e.g. valid JSON.

Many devs complain that #LLMs cannot be reliably used in production if the output is not valid, for instance, if one wants to use LLMs to generate SQL queries or JSON, it is crucial that the output is valid.

Frameworks have arisen to constrain the outputs of the LLM to follow some constraints, like outlines (https://github.com/outlines-dev/outlines), but they assume access to logits.

This makes them incompatible with proprietary LLMs like GPT4 that don’t share logits, so one can only use open-source LLMs that are much less performant.

This paper shows how can use powerful proprietary LLMs like GPT4 to create a first unconstrained sketch and refine it using an OSS model like Llama 2 where logits are accessible, to rewrite the sketch following some specific constraints.

They show that GPT4 Precision can be increased by 14% (43% before, 57% after), by boosting it with constrained output on information extraction on Wiki-NRE!


What's the best way to fine-tune open LLMs in 2024? Look no further! 👀 I am excited to share “How to Fine-Tune LLMs in 2024 with Hugging Face” using the latest research techniques, including Flash Attention, Q-LoRA, OpenAI dataset formats (messages), ChatML, Packing, all built with Hugging Face TRL. 🚀

It is created for consumer-size GPUs (24GB) covering the full end-to-end lifecycle with:
💡Define and understand use cases for fine-tuning
🧑🏻💻 Setup of the development environment
🧮 Create and prepare dataset (OpenAI format)
🏋♀ Fine-tune LLM using TRL and the SFTTrainer
🥇 Test and evaluate the LLM
🚀 Deploy for production with TGI

👉  https://www.philschmid.de/fine-tune-llms-in-2024-with-trl

Coming soon: Advanced Guides for multi-GPU/multi-Node full fine-tuning and alignment using DPO & KTO. 🔜

andrewrreed Jofthomas victor stefan-jo s3nh osanseviero luqmanxyz philschmid kramp samusenps Csplk lunarflu julien-c hamedbabaeigiglou harpreetsahota dhuynh95 clem CultriX afrideva brekk andysalerno PriyankSisodia azimuth3d nuatmochoi mathiasn1 ashwinlimaye AdinaY vicgalle pardeep merve adamo1139 madroid mikr omaryshchenko alielfilali01 patchie2 celsowm Luciano santyzenith sbrandeis ramkrish120595

BramVanroy clem osanseviero Norod78 hayoungpark azimuth3d mvonwyl pardeep KrishnaKaasyap merve lunarflu madroid zappa2005 Kamaljp mikr vivek37kumar raychen26 gzy02 InvictusRudra ddahlmeier