In Search of the Robust Facial Expressions Recognition Model: The Visual Cross-Corpus Study,  @ElenaRyumina  et al.

Model: https://huggingface.co/ElenaRyumina/face_emotion_recognition
Demo: https://huggingface.co/spaces/ElenaRyumina/Facial_Expression_Recognition
Repo: https://github.com/ElenaRyumina/EMO-AffectNetModel

Paper: https://www.sciencedirect.com/science/article/pii/S0925231222012656

@ElenaRyumina

osanseviero clem samusenps victor fffiloni gryhkn kramp


🙋🏻♂hey there folks ,

🤗Aya has been released !  It's an absolutely massive undertaking to create a huge multilingual dataset and multilingual model of very high quality.

Papers :
https://cohere.com/research/papers/aya-dataset-paper-2024-02-13
https://cohere.com/research/papers/aya-model-paper-2024-02-13

Model : https://huggingface.co/CohereForAI/aya-101
Dataset : https://huggingface.co/datasets/CohereForAI/aya_dataset

I am proud to be one of 3,000 humans who built Aya - a new massively multilingual, generative LLM that outperforms existing open-source models and covers 101 different languages. Together, we are accelerating multilingual AI. 🤗

tellarin mvonwyl marcvw-sightengine osanseviero davanstrien kramp jnemecek clem samusenps raghavprabhakar MexIvanov sarahooker alielfilali01 yjernite


Aya Dataset: An Open-Access Collection for Multilingual Instruction
  Tuning (2402.06619)

Aya Dataset

An Open-Access Collection for Multilingual Instruction Tuning

https://huggingface.co/papers/2402.06619

Datasets are foundational to many breakthroughs in modern artificial intelligence. Many recent achievements in the space of natural language processing (NLP) can be attributed to the finetuning of pre-trained models on a diverse set of tasks that enables a large language model (LLM) to respond to instructions. Instruction fine-tuning (IFT) requires specifically constructed and annotated datasets. However, existing datasets are almost all in the English language. In this work, our primary goal is to bridge the language gap by building a human-curated instruction-following dataset spanning 65 languages. We worked with fluent speakers of languages from around the world to collect natural instances of instructions and completions. Furthermore, we create the most extensive multilingual collection to date, comprising 513 million instances through templating and translating existing datasets across 114 languages. In total, we contribute four key resources: we develop and open-source the Aya Annotation Platform, the Aya Dataset, the Aya Collection, and the Aya Evaluation Suite. The Aya initiative also serves as a valuable case study in participatory research, involving collaborators from 119 countries. We see this as a valuable framework for future research collaborations that aim to bridge gaps in resources.

NeuralNovel alielfilali01 Pclanglais taufiqdp kargaranamir samusenps Dlbk macadeliccc osanseviero vumichien juancopi81 lhoestq davanstrien julien-c mathiasn1 clem raghavprabhakar sugatoray DataSoul Ryukijano grenwi

alielfilali01 osanseviero vumichien alvarobartt davanstrien julien-c clem Dlbk Ryukijano


Today I'm releasing marginalia, a python library to perform corpus analysis and retrieve structured annotations with open LLMs like Mistral Open-Hermes-2.5: https://github.com/Pleias/marginalia

marginalia leverages vllm inference speed to re-generate until all the output matches an expected json structure and to send batches of several unstructured elements for enhanced patterns detections. It works especially well for bibliographies. The demo transforms a very old list (Benjamin Franklin favorite's books from 1744) into well-structured data: https://colab.research.google.com/drive/1xKjK2mDDpXMaKG5YLpFhOM7jehxt0kEt?usp=sharing

While marginalia can be quite flexible, it definitely isn't a general purpose tool for json generation (like outlines). I don't intend so far to extend support to more complex json structure, but really looking forward potential feedbacks and suggestions.

davanstrien osanseviero merve AIIAR samusenps gate369 macadeliccc clem dillfrescott


Grandmaster-Level Chess Without Search (2402.04494)

Google released a paper on Chess that doesn't rely on MCTS (aka AlphaZero) ♟
their secret sauce is.. synthetic data pseudolabeled by Stockfish engine 😀
2024 really is the year of synthetic data across all domains!
There's a nice discussion here, join us https://huggingface.co/papers/2402.04494

alielfilali01 osanseviero AIIAR samusenps nickandbro Citaman clem cosmojg


🤗 Data is better together!

Data is essential for training good AI systems. We believe that the amazing community built around open machine learning can also work on developing amazing datasets together.

To explore how this can be done, Argilla and Hugging Face are thrilled to announce a collaborative project where we’re asking Hugging Face community members to build a dataset consisting of LLM prompts collectively.

What are we doing?
Using an instance of Argilla — a powerful open-source data collaboration tool — hosted on the Hugging Face Hub, we are collecting ratings of prompts based on their quality.

How Can You Contribute?
It’s super simple to start contributing:

1. Sign up if you don’t have a Hugging Face account

2. Go to this Argilla Space and sign in: https://huggingface.co/spaces/DIBT/prompt-collective

3. Read the guidelines and start rating prompts!

You can also join the #data-is-better-together channel in the Hugging Face Discord.

Finally, to track the community progress we'll be updating this Gradio dashboard:

https://huggingface.co/spaces/DIBT/prompt-collective-dashboard

davanstrien gabrielmbmb osanseviero ignacioct sdiazlor plaguss julien-c tomaarsen nickandbro victor lunarflu clem frascuchon abbanbhan lewtun merve samusenps MoritzLaurer johko lhoestq smangrul sayhan medmac01 maghwa

davanstrien gabrielmbmb osanseviero kramp ignacioct sdiazlor julien-c victor severo clem mvaloatto lewtun samusenps chkla smangrul Muttermal

davanstrien osanseviero sdiazlor julien-c victor clem smangrul


Created a collection with all the LLM hallucination and evaluation papers I've been reading over the last few weeks 📄📄

https://huggingface.co/collections/santiviquez/llm-hallucination-detection-papers-65c4d2399096960aa80776d3

Which one should I add to the list?

osanseviero clem gsarti merve samusenps alielfilali01


Model Editing with Canonical Examples (2402.06155)

🔍 Today's pick in Interpretability & Analysis of LMs: Model Editing with Canonical Examples by @johnhew @sachen @lora-x E. Adams P. Jiang @manning

This works introduces a model editing approach using individual “canonical” examples to showcase desired/unwanted behavior. An evaluation is then conducted on out-of-distribution samples spanning six datasets (3 introduced in this work) covering settings of interest in bias mitigation, hard syntactic constructions and knowledge-based predictions, while limiting the degradation of the original model’s loss.

Authors experiment with Pythia LMs, finding that LoRa fine-tuning on canonical examples outperforms other established editing methods such as MEMIT.

Then, the approach is tested on Backpack LMs, using a linear combination of sense vectors to disentangle semantic information in the input texts. In particular, authors introduce “sense fine-tuning” where only a handful of sense vectors is updated per example, which is shown to be more efficient yet more effective than regular fine-tuning.

Finally, the relation between the predictions of pre- and post-sense fine-tuning backpack LMs is used to successfully transfer the desired adaptation to a larger standard LM, at no performance cost.

📄 Paper: https://huggingface.co/papers/2402.06155

🔍 All daily picks in LM interpretability: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9


3 hours between the two pictures 🔥
Finally that paranoid inside me got some rest 😂

NeuralNovel Skier8402 Wauplin tomaarsen osanseviero clem merve macadeliccc 9voltfan2009


Hear, hear, AMD MI300Xs have started to emerge much sooner than expected.

Here is a 2-part benchmarks report on performing BLOOM-176B inference using @MSFTDeepSpeed optimized for AMD MI300X.

1. https://www.evp.cloud/post/diving-deeper-insights-from-our-llm-inference-testing
2. https://www.evp.cloud/post/diving-deeper-insights-from-our-llm-inference-testing-part-2

This was published in response to our BLOOM-176B super-fast inference blog post https://huggingface.co/blog/bloom-inference-pytorch-scripts

Note that these have 192GB of HBM!

The NVIDIA monopoly is strong, but it'll have to start sharing the pie and hopefully drive the costs down at least somewhat.

Thanks to https://www.linkedin.com/in/eliovp for sharing this writeup with me.

p.s. at the PyTorch conference in the fall, the AMD representative said we will see MI300X available to us mortals in Q4-2024/Q1-2025.

osanseviero julien-c davanstrien tomaarsen VictorSanh clem pcuenq muhtasham merve samusenps Eliovp andrewrreed rwightman clefourrier


Introducing Remove Background Web: In-browser background removal, powered by @briaai's new RMBG-v1.4 model and 🤗 Transformers.js!

Everything runs 100% locally, meaning none of your images are uploaded to a server! 🤯 At only ~45MB, the 8-bit quantized version of the model is perfect for in-browser usage (it even works on mobile).

Check it out! 👇
Demo: https://huggingface.co/spaces/Xenova/remove-background-web
Model: https://huggingface.co/briaai/RMBG-1.4

Felladrin osanseviero not-lain baotrinh samusenps mgfeller munirdin mvaloatto DamarJati AIIAR eskayML OriLib HDiffusion brianjking 3ginger ashishtanwer NeuralNovel taufiqdp kramp Waltert enzostvs victor merve nickandbro ecarbo raphaelmansuy Seuriin Renadi rreed-pha Noomam dillfrescott


Introducing UNA-SimpleSmaug-34b:

Based on Smaug-34B-v0.1, capable of slightly outperform his base model and with increased math and reasoning thanks to simple-math dataset.
The model exhibits a great performance across diverse tasks with an excellent and balanced behaviour.
It scores 77.41 AVG on the Leaderboard, landing on #1 Position of 34B models.

Available in the hub already:
https://huggingface.co/fblgit/UNA-SimpleSmaug-34b-v1beta
https://huggingface.co/datasets/fblgit/simple-math

In this case, we applied UNA to the Attention Layers of the model while performing SFT with simple-math on a high complexity generated data of mathematics, proving the effect of simple-math on LLM's.

fblgit samusenps macadeliccc osanseviero mrfakename Ji-Ha NeuralNovel taufiqdp merve linuxhackr


Repo with scripts to create your own moe models using Apple mlx is constantly updated by @mzbac here: https://github.com/mzbac/mlx-moe

It's an amazing resource to learn inner workings of lora on moe with mlx.
It uses https://huggingface.co/datasets/WizardLM/WizardLM_evol_instruct_70k as default dataset, but can be easily tweak to use any model or dataset fro HF

Have fun with it!

Pavarissy osanseviero awni NeuralNovel clem merve sugatoray


✨ In-context learning is all you need!

This super interesting paper shows that fine-tuning with #SFT or #RLHF only helps on the form but does not impact knowledge or reasoning abilities, and in some cases, actually decreases performance!

They tested it with Mistral-base vs Mistral FT-ed, as well as Llama 2 70b base and FT-ed and results are consistent.

Providing the right prompt to the base model actually makes the model better and has 0 training cost!

Paper: https://arxiv.org/abs/2312.01552

andysalerno samusenps osanseviero gl198976 VikramSingh178 akjindal53244 nickandbro clem victor anthonymikinka


🎉🥳🎉
Today, we are thrilled to officially launch the "2A2I" Arabic Artificial Intelligence Initiative. This is a community-driven initiative founded on the philosophy of "Small team, Big work" Our goal is to elevate Arabic AI (LLMs, Diffusion Models, ASR, etc.) to the same level as English (and also Chinese 🐉).

Naturally, our focus today is primarily on datasets. We aim to provide high-quality datasets, especially for LLMs this month, to support our future efforts. In line with this, we're excited to introduce the Arabic version of H4-no_robots, find here : https://huggingface.co/datasets/2A2I/H4_no_robots (and yes, we know it's not "no_robots" anymore 😄). Stay tuned for more exciting, high-quality datasets in the next couple of weeks (+ 4 million rows🔥)

In parallel, we're also developing a model 🐪 that we hope will set new high standards for Arabic LLMs. 🔥 This model is planned for release in the coming months.

For more information, please visit our Organization card here : https://huggingface.co/2A2I

If you're interested in Arabic AI and want to help pushing the wheel as well, fill out this form, and let us know your motivation and your exciting ideas 🔥

The form link : https://forms.gle/kZLVuynWFU2FyTm57

If you have any questions, feel free to reach out to us at the email address below.

Additionally, if you believe as we do in this mission and would like to help this community and contribute some compute resources 😉 or any other form of help you might think about, please contact us at the same email address below or reach out to me through LinkedIn 🔥

2A2I Contact Email : arabic.ai.initiative@gmail.com
My LinkedIn : https://www.linkedin.com/in/alielfilali01/

medmac01 samusenps macadeliccc HANTIFARAH andysalerno Rami osanseviero pain NeuralNovel kramp Kernel clem kenza-ily maghwa derek-thomas SaiedAlshahrani mahmoudtarek


The Truth is in There: Improving Reasoning in Language Models with
  Layer-Selective Rank Reduction (2312.13558)

Reducing perplexity in LLM's through layer selective rank reduction

Layer-Selective Rank Reduction (LASER) is a denoising method that improves reasoning by the strategic removal of higher-order components from weight matrices in the multi-layer perceptron (MLP) layers without the need for additional parameters or training data. This process leverages singular value decomposition to identify and eliminate these components. This simple, yet effective, method has shown to improve question-answering performance by up to 27.4 percentage points.

LaserRMT implements this through a process by calculating signal to noise ratio (SNR) for each layer and selectively reducing the rank of these layers.The SNR method meticulously computes the SNR by leveraging singular value decomposition (SVD) to separate the signal (higher-order components) from the noise (lower-order components) within the weight matrices of the model's layers. The SNR calculation is what determines which layers would benefit from rank reduction without compromising the models integrity.

If a layer is identified that could benefit from rank reduction, then the layer will enter an incremental process where the weight matrices are reduced and reconstructed by retaining only the singular values that surpass the threshold. In the case of laserRMT, the threshold is calculated by Marchenko-Pastur Law.
```
@staticmethod
    def marchenko_pastur_threshold(sigma, n, m):
        beta = n / m if n < m else m / n
        threshold = sigma * np.sqrt((1 + np.sqrt(beta))**2)
        return thr
```
The two primary benefits of applying this method are reducing computational overhead of large language models and simultaneously improving output quality.

Credit to @ehartford @fernandofernandes @DavidGF for laserRMT

Resources:
☄ AutoLaser: https://colab.research.google.com/drive/11j0e-w6BfvqeFN1gUrpOqdW0vcKqfVqP?usp=sharing
laserRMT: https://github.com/cognitivecomputations/laserRMT
https://huggingface.co/papers/2312.13558

samusenps DavidGF Undi95 mathiasn1 osanseviero KnutJaegersberg tiendung Ji-Ha Dlbk munirdin NeuralNovel victor Citaman abdullah


More Agents Is All You Need (2402.05120)

More Agents Is All You Need

https://huggingface.co/papers/2402.05120

find that, simply via a sampling-and-voting method, the performance of large language models (LLMs) scales with the number of agents instantiated. Also, this method is orthogonal to existing complicated methods to further enhance LLMs, while the degree of enhancement is correlated to the task difficulty. We conduct comprehensive experiments on a wide range of LLM benchmarks to verify the presence of our finding, and to study the properties that can facilitate its occurrence.


AttnLRP: Attention-Aware Layer-wise Relevance Propagation for
  Transformers (2402.05602)

🔍 Today's pick in Interpretability & Analysis of LMs: AttnLRP: Attention-Aware Layer-wise Relevance Propagation for Transformers by @RedOneAI et al.

This work proposes extending the LRP feature attribution framework to handling Transformers-specific layers. In particular: authors:

1. Propose a generalized approach to softmax linearization by designing a distribution rule that incorporates bias terms, absorbing of a portion of the relevance.
2. Propose decomposing the element-wise matrix multiplication in the attention operation as a sequential of epsilon and uniform distribution rules to ensure conservation (=sum of relevance stays constant across layers)
3. Propose handling normalisation layers with an identity distribution rule.

By means of extensive experiments, authors show that AttnLRP:

1. Is significantly more faithful than other popular gradient- and attention-based attribution approaches on CV and NLP tasks using large transformer models.
2. Runs in O(1) time, requiring O(sqrt(num_layers)) memory, as opposed to perturbation-based approaches requiring O(seq_len) time.
3. can be used alongside activation maximisation to explain the contribution of granular model components in driving models’ predictions.

📄 Paper: https://huggingface.co/papers/2402.05602

🔍 All daily picks in LM interpretability: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9

osanseviero RedOneAI erfanhatefi alielfilali01 OctoOptLab


We have recently shipped two new task pages thanks to the contributors!
Both of these pages consists of models you can use off-the-shelf 💜
Reading these you can start building in no time 🛠
Task page for zero-shot object detection 👉  https://huggingface.co/tasks/zero-shot-object-detection 📚
Task page for mask generation 👉  https://huggingface.co/tasks/mask-generation 📖

fffiloni kramp osanseviero macadeliccc AdinaY not-lain nbroad

jeffboudier santiviquez macadeliccc not-lain alielfilali01


GlotLID: Language Identification for Low-Resource Languages (2310.16248)

A Text Language Identification Model with Support for +2000 Labels:

space: https://huggingface.co/spaces/cis-lmu/glotlid-space
model: https://huggingface.co/cis-lmu/glotlid
github: https://github.com/cisnlp/GlotLID
paper: https://huggingface.co/papers/2310.16248

s3nh tomaarsen merve kardal Sylvestre kramp osanseviero macadeliccc kaizuberbuehler FPK39 lbourdois DmitryRyumin sbarman25 ayymen


The most widely used French NER models on HF (https://huggingface.co/Jean-Baptiste/camembert-ner and https://huggingface.co/cmarkea/distilcamembert-base-ner) are trained on a single dataset (WikiNER) which on the one hand contains leaks and therefore distorts the true results of these models, and on the other hand overspecializes them in a particular domain (= texts from Wikipedia). They are also only available in a base version (110M parameters).

That's why I've trained new NER models in French both on more data (x3), as well as in base and large versions (336M). They are available in 3 entities (PER, ORG, LOC) or 4 entities (PER, ORG, LOC, MISC):
- https://huggingface.co/CATIE-AQ/NERmembert-base-4entities
- https://huggingface.co/CATIE-AQ/NERmembert-large-4entities
- https://huggingface.co/CATIE-AQ/NERmembert-base-3entities
- https://huggingface.co/CATIE-AQ/NERmembert-large-3entities

Datasets without leaks are also available:
- https://huggingface.co/datasets/CATIE-AQ/frenchNER_4entities
- https://huggingface.co/datasets/CATIE-AQ/frenchNER_3entities

osanseviero Tonic EquinoxElahin mishig tomaarsen


🚀 It's now easier than ever to switch from OpenAI to open LLMs

Hugging Face's TGI now supports an OpenAI compatible Chat Completion API

This means you can transition code that uses OpenAI client libraries (or frameworks like LangChain 🦜  and LlamaIndex 🦙) to run open models by changing just two lines of code 🤗

⭐ Here's how:
```
from openai import OpenAI

# initialize the client but point it to TGI
client = OpenAI(
    base_url="<ENDPOINT_URL>" + "/v1/",  # replace with your endpoint url
    api_key="<HF_API_TOKEN>",  # replace with your token
)
chat_completion = client.chat.completions.create(
    model="tgi",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Why is open-source software important?"},
    ],
    stream=True,
    max_tokens=500
)

# iterate and print stream
for message in chat_completion:
    print(message.choices[0].delta.content, end="")
```

🔗 Blog post ➡ https://huggingface.co/blog/tgi-messages-api
🔗 TGI docs ➡ https://huggingface.co/docs/text-generation-inference/en/messages_api

osanseviero clem samusenps ashesashes johko Tonic julien-c victor Chunte johnnybio Alisherfullhd Yogeek Glavin001 nbroad Nymbo

samusenps hunkim kramp Tonic julien-c Chunte jeffboudier natolambert Felladrin Nymbo

mvaloatto hunkim Tonic julien-c Glavin001 Nymbo


First HF social post:

pip install -U mlx

osanseviero clem ivanfioravanti multimodalart johko merve victor reach-vb hfthorvaldur flymonk

osanseviero clem multimodalart merve kardal reach-vb


Apple MLX 0.2.0 is here!

@awni and his team did again! Mega release.

Performance 💨💨💨  is coming to town!
mx.compile makes stuff go fast
- Some functions are up to 10x faster (benchmarks: https://github.com/ml-explore/mlx/pull/614#issuecomment-1929056286)
- Training models anywhere from 10% to twice as fast (benchmarks: https://github.com/ml-explore/mlx-examples/pull/420#issuecomment-1932422605)

Full details on this release here: https://github.com/ml-explore/mlx/releases/tag/v0.2.0

awni osanseviero clem heyyeah samusenps akashicmarga merve victor kroonen Gatozu35


Benefits of `imatrix` quantization in place of quip#

Quip-# is a quantization method proposed by [Cornell-RelaxML](https://github.com/Cornell-RelaxML) that claims tremendous performance gains using only 2-bit precision.

RelaxML proposes that quantizing a model from 16 bit to 2 bit precision they can utilize Llama-2-70B on a single 24GB GPU.

QuIP# aims to revolutionize model quantization through a blend of incoherence processing and advanced lattice codebooks. By switching to a Hadamard transform-based incoherence approach, QuIP# enhances GPU efficiency, making weight matrices more Gaussian-like and ideal for quantization with its improved lattice codebooks.

This new method has already seen some adoption by projects like llama.cpp. The use of the Quip-# methodology has been implemented in the form of imatrix calculations. The importance matrix is calculated from a dataset such as wiki.train.raw and will output the perplexity on the given dataset.

This interim step can improve the results of the quantized model. If you would like to explore this process for yourself:

llama.cpp - https://github.com/ggerganov/llama.cpp/
Quip# paper - https://cornell-relaxml.github.io/quip-sharp/
AutoQuip# colab -  https://colab.research.google.com/drive/1rPDvcticCekw8VPNjDbh_UcivVBzgwEW?usp=sharing

Other impressive quantization projects to watch:
+ AQLM
https://github.com/Vahe1994/AQLM
https://arxiv.org/abs/2401.06118

nivu clem merve osanseviero ybelkada samusenps NeuralNovel Ji-Ha hllj DamarJati sayhan NovoCode sugatoray 1TuanPham victor John6666 louisbrulenaudet

clem merve osanseviero ybelkada NeuralNovel Gatozu35 akjindal53244 sayhan NovoCode KingNish


Corrective Retrieval Augmented Generation (2401.15884)

What if the retrieval goes wrong? 🐕

Retrieval Augmented Generation (RAG) is a strategy to alleviate LLM hallucinations and improve the quality of generated responses.

A standard RAG architecture has two main blocks: a Retriever and a Generator.

1⃣ When the system receives an input sequence, it uses the Retriever to retrieve the top-K most relevant documents associated with the input sequence. These documents typically come from an external source (e.g., Wikipedia) and are then concatenated to the original input's context.

2⃣ It then uses the Generator to generate a response given the gathered information in the first step.

But what happens if the retrieval goes wrong and the retrieved documents are of very low quality?

Well, in such cases, the generated response will probably be of low quality, too. 🫠

But here is where CRAG (Corrective RAG) *might* help. I say it might help because the paper is very new — only one week old, and I don't know if someone has actually tried this in practice 😅

However, the idea is to add a Knowledge Correction block between the Retrieval and Generation steps to evaluate the retrieved documents and correct them if necessary.

This step goes as follows:

🟢 If the documents are correct, they will be refined into more precise knowledge strips and concatenated to the original context to generate a response.

🔴 If the documents are incorrect, they will be discarded, and instead, the system searches the web for complementary knowledge. This external knowledge is then concatenated to the original context to generate a response.

🟡 If the documents are ambiguous, a combination of the previous two resolutions is triggered.

The experimental results from the paper show how the CRAG strategy outperforms traditional RAG approaches in both short and long-form text generation tasks.

Paper: https://huggingface.co/papers/2401.15884

osanseviero samusenps NeuralNovel nickandbro abellion


Grandmaster-Level Chess Without Search

https://huggingface.co/papers/2402.04494

largest model reaches a Lichess blitz Elo of 2895 against humans, and successfully solves a series of challenging chess puzzles, without any domain-specific tweaks or explicit search algorithms. We also show that our model outperforms AlphaZero's policy and value networks (without MCTS) and GPT-3.5-turbo-instruct. A systematic investigation of model and dataset size shows that strong chess performance only arises at sufficient scale. To validate our results, we perform an extensive series of ablations of design choices and hyperparameters.

samusenps andysalerno osanseviero NeuralNovel


Prompts are hyperparameters. Every time you test a different prompt on your data, you become less sure if the LLM actually generalizes to unseen data.

Issues of overfitting to a test set seem like concepts from boring times when people still fine-tuned models, but it's just as important for "zeroshot prompting". Using a separate validation split to tune the main hyperparameter of LLMs (the prompt) is just as important as train-val-test splitting for fine-tuning. The only difference is that you don't have a training dataset anymore and it somehow feels different because there is no training / no parameter updates.

Its easy to trick yourself into believing that an LLM performs well on your task, while you've actually overfit the prompt on your data. Every good "zeroshot" paper should clarify that they used a validation split for finding their prompt before final testing.

samusenps tollefj merve osanseviero sayhan


Hi friends, i'am happy to share with you all a tool that i built a week ago or so, i'am talking here about the "LLM Training Cost Calculator" - a handy tool now available on Hugging Face Spaces! This interactive Gradio app provides an easy-to-use interface for estimating the training costs of large language models (LLMs).

(I've been asked to provide a report about the cost of finetuning each model etc... so i decided to do the lazy job and build a tool for it, Prof later can choose whatever config he likes 😆)

🔍 But Why this is important?
As LLMs continue to grow in size and complexity, understanding the computational and financial requirements is crucial for planning and managing AI projects. I believe this tool simplifies this process, giving you insights into potential expenses based on the number of parameters and tokens in your dataset.

🌟 Features:
- Input the number of parameters (in billions) and tokens (in trillions).
- Adjust for GPU utilization rates and overhead costs.
- Get an instant estimate of your training costs.
+ Choose your GPU (A100 80GB PCle, A100 80GB SXM, V100, H100 SXM, H100 PCle)

📈 Coming Soon:
Plans are in place to expand the calculator's capabilities to include fine-tuning costs for models using LoRA or QLoRA. You'll be able to input a model ID from the Hugging Face Hub, select your fine-tuning strategy, and specify quantization details if using QLoRA.

I believe this tool will be a valuable asset to the AI community, helping to plan and allocate resources more effectively 🤗.

Should you have any suggestions or feedback, please don't hesitate to contribute your thoughts in the comments below. Together, we can refine and enhance this resource for all.

🔗 Try it here : https://huggingface.co/spaces/Ali-C137/LLM-Training-Cost-Calculator

PS : All thanks to Gradio, Hugging Face and the community ofc 🔥 😉

victor merve Sylvestre osanseviero ivanfioravanti


A while ago, I presented this Phi2 DPO fine-tune notebook with LoRa. Got some input from @ybelkada  about not needing a `ref_model` because we can just swap out the LoRa adapters during training. Cool feature 🤓

https://colab.research.google.com/drive/1PGMj7jlkJaCiSNNihA2NtpILsRgkRXrJ#scrollTo=wXqoH2TMnjjp

merve samusenps ybelkada alielfilali01 sumandas compressionsavant wise-east


Welcome Bunny! A family of lightweight but powerful multimodal models from BAAI

With detailed work on dataset curation, the Bunny-3B model built upon SigLIP and Phi-2 achieves performance on par with 13B models.

Model: https://huggingface.co/BAAI/bunny-phi-2-siglip-lora


There appears to be a huge misunderstanding regarding the licensing requirements for open sourced Chinese speaking speaking LLMs on
@huggingface

I initially shared this misconception too, but after conducting some research, I came up with the list below.

Veryimpressive!

osanseviero samusenps AdinaY Tonic yahma Tom9000 wwizo aloobun sayhan


🌠 Let's try to figure out this one as a community.
What reactions should we add to Posts and discussions?

If the reaction you want is already in the replies give it a thumbs up (👍) if it's not just add it as a reply.

julien-c Sylvestre santiviquez kramp taufiqdp osanseviero ArthurZ samusenps clem mvaloatto Chunte


Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations
  from Large Language Models (2402.04614)

🔍 Today's pick in Interpretability & Analysis of LMs: Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Models by C. Agarwal, S.H. Tanneru and H. Lakkaraju

This work discusses the dichotomy between faithfulness and plausibility in LLMs’ self-explanations (SEs) in natural language (CoT, counterfactual reasoning, and token importance). These explanations tend to be reasonable according to human understanding (plausible) but are not always aligned with the reasoning processes of the LLMs (unfaithful).

Authors remark that the increase in plausibility driven by the request for a friendly conversational interface might come at the expense of faithfulness. Provided the faithfulness requirements of many high-stakes real-world settings, authors suggest these are considered when designing and evaluating new explanation methodologies.

Finally, the authors call for a community effort to 1) develop reliable metrics to characterize the faithfulness of explanations and 2) pioneering novel strategies to generate more faithful SEs.

📄 Paper: https://huggingface.co/papers/2402.04614

🔍 All daily picks in LM interpretability: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9


#lg #gram #solarllm BREAKING NEWS:

upstage/Solar LLM will soon be available for LG gram Laptops as an on-device LLM. 💻🌞🎉

Upstage makes LLMs accessible to everyone and every device. We'd love to see more on-device LLMs.

https://koreajoongangdaily.joins.com/news/2024-02-06/business/industry/LG-Electronics-signs-partnership-with-generative-AI-startup-Upstage-/1975528

samusenps leonardlin osanseviero hf-delta akashicmarga


From @IkariDev and @Undi95
New release from NeverSleep!

MiquMaid-v2-70B
https://huggingface.co/NeverSleep/MiquMaid-v2-70B
https://huggingface.co/NeverSleep/MiquMaid-v2-70B-GGUF
DPO version
https://huggingface.co/NeverSleep/MiquMaid-v2-70B-DPO
https://huggingface.co/NeverSleep/MiquMaid-v2-70B-DPO-GGUF

MiquMaid-v2-2x70B
https://huggingface.co/NeverSleep/MiquMaid-v2-2x70B
https://huggingface.co/NeverSleep/MiquMaid-v2-2x70B-GGUF
DPO version
https://huggingface.co/NeverSleep/MiquMaid-v2-2x70B-DPO
https://huggingface.co/NeverSleep/MiquMaid-v2-2x70B-DPO-GGUF

Feedback appreciated!

uncensorie frammie hunkim samusenps s3nh Waltert not-lain rewij okeksama IHaBiS


Are you up to a 🤗 challenge 🏆 ?
if so 👀 Check out the new MindBigData Leaderboard 🔥🔥🔥
🚀 https://huggingface.co/spaces/DavidVivancos/MindBigData-Leaderboard

Decode the "source" 🧠 with the largest multimodal opendata of brain signals for Machine Learning.

Try to beat the whopping 🥇98,97% accuracy of Smita Tiwari Shivani Goel School of CSET Bennett University, India and Arpit Bhardwaj BML Munjal University decoding the multiclass Yann LeCun mnist of brain digits caputured with EMOTIV Epoc and 🥇89,62% with Insight

Or the 🥇96,18% of Dr. Nrushingh Charan Mahapatra Intel Corporation and Prof.(Dr). Prachet Bhuyan Kalinga Institute of Industrial Technology, Bhubaneswar also with the mnist of brain digits but captured with Muse® by Interaxon Inc.

Or the 🥇85% of Matthew Zhang Westlake High School and Jeremy Lu now at Purdue University decoding brain images captured from imagenet

Or be the first to break the 🧊 with the largest open dataset 2023 (8+ billion datapoints), the multimodal MindBigData2023_MNIST-8B captured with a custom 128 channels EEG that I built and with the real 70,000 MNIST digits and put your NVIDIA gpus to work.

All the datasets are open and ready at HuggingFace, dare to try?

Hope to see you all soon in the LeaderBoard

Thanks
@DavidVivancos

TF1920 osanseviero clem samusenps victor julien-c clefourrier Tonic


EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameters (2402.04252)

EVA-CLIP 🦖 is the CLIP scaled to the moon! 🔥
The new SotA CLIP-like model 🏆
Highlights ✨
- Performs better in linear probing
- Outperforms in Zero-Shot Image-Text Retrieval
- Higher zero-shot accuracy in IN-1K

As usual, try it with the notebook I built for you https://colab.research.google.com/drive/1K7DdCORC3x4qyhwhuB4fT4wcfJ_BQLKw?usp=sharing#scrollTo=0ZS_lJ7SK6Ys
I also built a Space for you to compare the output probabilities to CLIP, seems that EVACLIP is more "sure" of it's results 😊 https://huggingface.co/spaces/merve/EVACLIP
The authors have shared 8B checkpoints open with Apache 2.0 license 💜 and it's built on top of transformers, super easy to use! https://huggingface.co/BAAI/EVA-CLIP-8B
Read the paper https://huggingface.co/papers/2402.04252 📄

victor osanseviero santiviquez clem AIIAR samusenps danielus boapps qiying


Super excited to share my project, ageML, here! 😊

ageML is a Python library I've been building to study the temporal performance degradation of ML models.

The goal of the project is to facilitate the exploration of performance degradation by providing tools for people to easily test how their models would evolve over time when trained and evaluated on different subsets of their data.

⭐ Check it out: https://github.com/santiviquez/ageml

victor osanseviero kramp clem hunkim samusenps merve abhishekvarma12345 MarinaraSpaghetti eevee32x


Self-Discover: Large Language Models Self-Compose Reasoning Structures (2402.03620)

Self-Discover

Large Language Models Self-Compose Reasoning Structures

paper page: https://huggingface.co/papers/2402.03620

SELF-DISCOVER substantially improves GPT-4 and PaLM 2's performance on challenging reasoning benchmarks such as BigBench-Hard, grounded agent reasoning, and MATH, by as much as 32% compared to Chain of Thought (CoT). Furthermore, SELF-DISCOVER outperforms inference-intensive methods such as CoT-Self-Consistency by more than 20%, while requiring 10-40x fewer inference compute. Finally, we show that the self-discovered reasoning structures are universally applicable across model families: from PaLM 2-L to GPT-4, and from GPT-4 to Llama2, and share commonalities with human reasoning patterns.

osanseviero Erland clem vaibhavgeek merve akashicmarga


Memphis: Advancing language model reasoning without relying on proprietary model outputs

Memphis is a series of models which advance human-data models, offering good performance without relying on proprietary model outputs (e.g. GPT-generated datasets). I've developed a new iterative finetuning procedure to improve the reasoning ability of these models beyond what is possible using only SFT on the same data.

Currently, I've released two models: Memphis-CoT-3B, and Memphis-scribe-3B.

To create these models, I've created new datasets:
- https://huggingface.co/datasets/euclaise/reddit-instruct : A dataset of instruction/QA-like data scraped from Reddit. A curated version, filtered using Lilac and neural embedding models, is available at https://huggingface.co/datasets/euclaise/reddit-instruct-curated
- https://huggingface.co/datasets/euclaise/TinyCoT : TinyCoT is a mtea-dataset that aggregates a variety of different human-sourced reasoning data. It is a curated version of my previous MegaCoT dataset https://huggingface.co/datasets/euclaise/MegaCoT, which contains 629k responses which get cut down to 28k for TinyCoT.  There's also an intermediate version https://huggingface.co/datasets/euclaise/MiniCoT, which has 129k responses.

Memphis-CoT is trained on reddit-instruct, a filtered version of oasst2 https://huggingface.co/datasets/sablo/oasst2_curated, and TinyCoT. Multiple iterations were performed on TinyCoT, while reddit-instruct and oasst2 were only used for the initial model.

Memphis-scribe further finetunes Memphis-CoT on more creative tasks. It was finetuned from Memphis-CoT on 18 different datasets, including datasets like https://huggingface.co/datasets/euclaise/WritingPrompts_curated, https://huggingface.co/datasets/lemonilia/LimaRP, and more.

To prevent catastrophic forgetting, I used weight averaging between iterations.

- https://huggingface.co/euclaise/Memphis-CoT-3B
- https://huggingface.co/euclaise/Memphis-scribe-3B

osanseviero lbourdois victor samusenps xsa-dev merve ABIN1012 d0rj alielfilali01 reciprocate HailJebus qqqzzzyyy


INSIDE: LLMs' Internal States Retain the Power of Hallucination
  Detection (2402.03744)

🔍 Today's pick in Interpretability & Analysis of LMs: INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection by @chaochen et al.

Previous efforts in detecting hallucinations using model intrinsic information employed predictive uncertainty or self-consistency to detect evaluation. Authors contend that in these procedure the rich semantic information captured in model embeddings is inevitably lost while decoding tokens.

To prevent this information loss they propose EigenScore, an internal measure of responses’ self-consistency using the eigenvalues of sampled responses' covariance matrix in intermediate model layers to quantify answers’ diversity in the dense embedding space.

Results show that EigenScore outperforms logit-level methods for hallucination detection on QA tasks, especially when paired with inference time feature clipping to truncate extreme activations, reducing overconfident generations.

📄 Paper: https://huggingface.co/papers/2402.03744


Introducing Vision Arena (beta)! Based on the lmsys's ChatbotArena, we create a simple demo for testing different Vision LMs (VLMs). We now  support GPT-4V, Gemini-Pro-Vision, and Llava.  More updates and models will come soon! We are still in the development stage and for now and we'd love to hear your feedback and suggestions! Please help us vote for better VLMs in your own use cases here! :D  Kudos to Yujie Lu (UCSB)!
https://huggingface.co/spaces/WildVision/vision-arena

merve yuchenlin Dlbk victor osanseviero lewtun clefourrier multimodalart FunCube JustinLin610 VictorSanh drogozhang samusenps tdambrowitz AdinaY natolambert vishaal27 MoritzLaurer taesiri


We've been busy cooking up some interesting models at @jinaai, with a recent highlight being the release of our first batch of bilingual embedding models.

Internally labeled as `X+EN`, where X represents the target language and `EN` stays fixed, these models specialize in both monolingual tasks and cross-lingual retrieval tasks, crossing from X to EN.

You can find these models available on Huggingface:
1. German-English bilingual embedding: https://huggingface.co/jinaai/jina-embeddings-v2-base-de
2. Chinese-English bilingual embedding: https://huggingface.co/jinaai/jina-embeddings-v2-base-zh

We're also excited to announce that a Spanish bilingual embedding will be released in approximately two weeks.

Our evaluation across various MLM tasks has demonstrated that the Bilingual Backbone consistently outperforms state-of-the-art Multilingual Backbones like XLM-Roberta (given its focus on just two languages).

Despite being three times smaller than the leading multilingual models (e5-multilingual-large), our released bilingual embedding models have shown superior performance compared to e5-multilingual-large, excelling in both monolingual and cross-lingual search tasks.

Currently, we're putting the finishing touches on the technical report, which should be available on Arxiv by next week.

Looking ahead, the embedding team is gearing up for `jina-embeddings-v3`
 with some initial groundwork already underway. Stay tuned for more updates!

KnutJaegersberg merve Dlbk osanseviero tomaarsen FremyCompany victor


Yesterday we just released Qwen1.5. Maybe someday I can tell more about the experience. But this is is at least a good release even if it is not yet SOTA. There is not so many SOTA by the way. This time, we actually fixed a lot of problems.

1. Context lengths are finally unified for all sizes. Previously, a lot of users kept telling us that 14B only supports 2K (Yeah even dynamic NTK does not work that well and it can only be extended to around 4-5K. Let alone those know nothing about how to use dynamic NTK).

2. If you carefully use our base language models, you will find that they understand special tokens of ChatML, which means that you can directly use LoRA to train on data with ChatML format. Why you can't do this before? This is because if the base language model does not understand the special tokens, you need to make them trained, which means that you should turn on the training of embedding. This is disgusting and it often leads to problems when you use ZeRO3.

3. We did strengthen our base language models except for 72. You should find better base language models, especially for 7 and 14. Why not 72? Nah, hard to say, but will make it better.

4. About the multilingual capabilities. Yes we finally build up our multilingual evaluation system and find out that our new base language models have nice  performance in multilingual evaluation for base language models. This tells us that we should pay more attention to the post-training with multilingual data. And we did that too. This is why this time we tell you something about multilingual performance. It is for sure much much better than our models before this release.

5. Chat models are the most promising stuff. Before this release, we gave you the SFT models. But this time, we had very nice SFT+DPO models. Yeah not only annotators like them but also users like them. I am sure you developers will feel that way too.

s3nh osanseviero Pclanglais andysalerno hf-delta huodon KnutJaegersberg cin-hubert xu3kev hiyouga merve Stopwolf Dlbk ShushengYang lewtun Ji-Ha xianbao euclaise smerchi AdinaY Sangmin philschmid joaogante victor nateraw dlimeng bowiehsu jkeisling HDiffusion EditaZ

osanseviero merve xianbao AdinaY philschmid


DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open
  Language Models (2402.03300)

DeepSeekMath

Pushing the Limits of Mathematical Reasoning in Open Language Models

paper page: https://huggingface.co/papers/2402.03300

DeepSeekMath 7B has achieved an impressive score of 51.7% on the competition-level MATH benchmark without relying on external toolkits and voting techniques, approaching the performance level of Gemini-Ultra and GPT-4

clem ajibawa-2023 osanseviero merve Dlbk andreinigo distantquant hunkim alielfilali01 Sattineez mysticaltech


BARTScore: Evaluating Generated Text as Text Generation (2106.11520)

Understanding BARTScore 🛹

BARTScore is a text-generation evaluation metric that treats model evaluation as a text-generation task 🔄

Other metrics approach the evaluation problem from different ML task perspectives; for instance, ROUGE and BLUE formulate it as an unsupervised matching task, BLUERT and COMET as a supervised regression, and BEER as a supervised ranking task.

Meanwhile, BARTScore formulates it as a text-generation task. Its idea is to leverage BART's pre-trained contextual embeddings to return a score that measures either the faithfulness, precision, recall, or F-score response of the main text-generation model.

For example, if we want to measure faithfulness, the way it works is that we would take the source and the generated text from our model and use BART to calculate the log token probability of the generated text given the source; we can then weight those results and return the sum.

BARTScore correlates nicely with human scores, and it is relatively simple to implement.

📑 Here is the original BARTScore paper: https://huggingface.co/papers/2106.11520
🧑💻 And the GitHub repo to use this metric: https://github.com/neulab/BARTScore


Rethinking Interpretability in the Era of Large Language Models (2402.01761)

🔍 Today's pick in Interpretability & Analysis of LMs: Rethinking Interpretability in the Era of Large Language Models
by C. Singh, J. P. Inala, @mgalley, R. Caruana, @wyngjf

In this opinion piece, authors contend that the new capabilities of LLMs can deeply transform the scope of interpretability, moving from low-level explanations such as saliency maps to natural language explanations that would allow for natural interaction with users.

This ambitious goal is however hindered by LM’s natural tendency to hallucinate, their large size and their inherent opaqueness. Authors highlight in particular dataset explanations for knowledge discovery, explanations’ reliability and interactive explanations as important priorities for the future of interpretability research.

📄 Paper: https://huggingface.co/papers/2402.01761


Today my RunPod pod was broken and I didn't notice until I fully did setup it. So I have written the following tutorial for how to deploy a Pod and also verify it is not broken.

You can read on:

Patreon (public) : https://www.patreon.com/posts/how-to-deploy-on-97919576

Medium (public) : https://medium.com/@furkangozukara/how-to-deploy-a-pod-on-runpod-and-verify-it-is-working-20e47031c0b5

CivitAI (public) : https://civitai.com/articles/3994/how-to-deploy-a-pod-on-runpod-and-verify-it-is-working

LinkedIn (public) : https://www.linkedin.com/pulse/how-deploy-pod-runpod-verify-working-furkan-g%2525C3%2525B6z%2525C3%2525BCkara-lgplf%3FtrackingId=EuNOjpKCSQ%252BVfpiQV3D6KQ%253D%253D/?trackingId=EuNOjpKCSQ%2BVfpiQV3D6KQ%3D%3D

Dev . to (public) : https://dev.to/furkangozukara/how-to-deploy-a-pod-on-runpod-and-verify-it-is-working-3pop


StepCoder: Improve Code Generation with Reinforcement Learning from
  Compiler Feedback (2402.01391)

StepCoder

Improve Code Generation with Reinforcement Learning from Compiler Feedback

paper page: https://huggingface.co/papers/2402.01391

The advancement of large language models (LLMs) has significantly propelled the field of code generation. Previous work integrated reinforcement learning (RL) with compiler feedback for exploring the output space of LLMs to enhance code generation quality. However, the lengthy code generated by LLMs in response to complex human requirements makes RL exploration a challenge. Also, since the unit tests may not cover the complicated code, optimizing LLMs by using these unexecuted code snippets is ineffective. To tackle these challenges, we introduce StepCoder, a novel RL framework for code generation, consisting of two main components: CCCS addresses the exploration challenge by breaking the long sequences code generation task into a Curriculum of Code Completion Subtasks, while FGO only optimizes the model by masking the unexecuted code segments to provide Fine-Grained Optimization. In addition, we furthermore construct the APPS+ dataset for RL training, which is manually verified to ensure the correctness of unit tests. Experimental results show that our method improves the ability to explore the output space and outperforms state-of-the-art approaches in corresponding benchmarks.


Necessity is the mother of invention, and of Gradio components.

Sometimes we realize that we need a Gradio component to build a cool application and demo, so we just build it. For example, we just added a new gr.ParamViewer component because we needed it to display information about Python & JavaScript functions in our documentation.

Of course, our users should be able able to do the same thing for their machine learning applications, so that's why Gradio lets you build custom components, and publish them to the world 🔥

sbarman25 victor samusenps ysharma ajibawa-2023 gsarti osanseviero dvilasuero hmb notsahil


These past months, I've been busy baking a special sort of Croissant 🥐 with an awesome team !

🥐 CroissantLLM is a truly bilingual language model trained on 3 trillion tokens of French and English data. In its size category (<2B), it is the best model in French, but it also rivals the best monolingual English models !

💾 To train it, we collected, filtered and cleaned huge quantities of permissively licensed French data, across various domains (legal, administrative, cultural, scientific), and different text modalities (speech transcriptions, movie subtitles, encyclopedias, forums, webpages)...

⚖ Assessing  LLM performance is not easy, especially outside of English, and to this end we crafted a novel evaluation benchmark, FrenchBench, aiming to assess reasoning, factual knowledge, and linguistic capabilities of models in French !

🔎 The best current LLMs are hidden behind a shroud of mystery, trained with undisclosed training data mixes or strategies. We go the opposite way, releasing all of the project's artefacts (model checkpoints, data, training details, evaluation benchmarks...) We obtain 81 % of the Stanford FMTI transparency criterias, far ahead of even most open initiatives !

🧪Beyond a powerful industrial resource, our transparent initiative is a stepping stone for many scientific questions ! How does teaching a model two languages instead of one splits its monolingual ability ? Does training on so much French help the model integrate French-centric knowledge and cultural biases ? How does the model memorize the training data ?

Many more things to say, for those interested, I recommend checking out:

🗞 The blogpost: https://huggingface.co/blog/manu/croissant-llm-blog
📖 The 45 page report with lots of gems: https://arxiv.org/abs/2402.00786
🤖 Models, Data, Demo: https://huggingface.co/croissantllm

eliolio gsarti euclaise eliebak kramp yozozaya clem blanchon victor mvaloatto osanseviero samusenps santiviquez ksiabani batmac nouamanetazi davanstrien fffiloni alielfilali01 sbrandeis Soubz velaia


Looking for a Needle in a Haystack: A Comprehensive Study of
  Hallucinations in Neural Machine Translation (2208.05309)

SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for
  Generative Large Language Models (2303.08896)

LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain
  Conversations with Large Language Models (2305.13711)

Some of my results from experimenting with hallucination detection techniques for LLMs 🫨🔍

First, the two main ideas used in the experiments—using token probabilities and LLM-Eval scores—are taken from these three papers:

1. https://huggingface.co/papers/2208.05309
2. https://huggingface.co/papers/2303.08896
3. https://huggingface.co/papers/2305.13711

In the first two, the authors claim that computing the average of the sentence-level token probabilities is the best heuristic for detecting hallucinations. And from my results, we do see a weak positive correlation between average token probabilities and ground truth. 🤔

The nice thing about this method is that it comes with almost no implementation cost since we only need the output token probabilities from the generated text, so it is straightforward to implement.

The third paper proposes an evaluation shema where we do an extra call to an LLM and kindly ask it to rate on a scale from 0 to 5 how good the generated text is on a set of different criteria. 📝🤖

I was able to reproduce similar results to those in the paper. There is a moderate positive correlation between the ground truth scores and the ones produced by the LLM.

Of course, this method is much more expensive since we would need one extra call to the LLM for every prediction that we would like to evaluate, and it is also very sensitive to prompt engineering. 🤷

osanseviero victor clem samusenps gsarti KasperNomm JasperV13 sbrandeis


A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for
  Verifiers of Reasoning Chains (2402.00559)

🔍 Today's pick in Interpretability & Analysis of LMs: A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains by @alonjacovi @yonatanbitton B. Bohnet J. Herzig @orhonovic M. Tseng M. Collins @roeeaharoni @mega

This work introduces a new methodology for human verification of reasoning chains and adopts it to annotate a dataset of chain-of-thought reasoning chains produced by 3 LMs. The annotated dataset, REVEAL, can be used to benchmark automatic verifiers of reasoning in LMs.

In their analysis, the authors find that LM-produced CoTs generally contain faulty steps, often leading to incorrect automatic verification. In particular, CoT-generating LMs are found to produce non-attributable reasoning steps often, and reasoning verifiers generally struggle to verify logical correctness.

📄 Paper: https://huggingface.co/papers/2402.00559
🔡 Dataset: https://huggingface.co/datasets/google/reveal

s3nh roeeaharoni alonjacovi osanseviero clem manu


📣 DPO Dutch model release + datasets

After teasing for a while, I am finally releasing **GEITje 7B Ultra**, building upon the great GEITje 7B by @Rijgersberg. New contributions include: large new datasets for SFT (instruction/chat), two datasets for DPO training (i.e. RLAIF), and an SFT and DPO version of GEITje. The READMEs describe everything well (I hope), and I'll also share more info on social medias tomorrow.

For me this is a huge release, the datasets more so than the models. I'm especially pleased with UltraChat, which I created with the intent of having a diverse dataset - the model must be able to communicate with different types of users. So the user questions are created as if they were written by different personas, e.g. language learners, young children, experts, critics, etc. The focus with this is "building a good communication bot that is accessible and can handle different kinds of user input".

I wish I could find the time to also write a paper to get some "academic recognition" but that'll have to wait for now. I just want to bring it to the public so that others can play with it and use it to build new, cool stuff!

I hope that you can all appreciate the work. Let's build some cool stuff with it!

Models:
- Demo: https://huggingface.co/spaces/BramVanroy/GEITje-7B-ultra
- DPO Model: https://huggingface.co/BramVanroy/GEITje-7B-ultra
- SFT model (not recommended): https://huggingface.co/BramVanroy/GEITje-7B-ultra-sft

Datasets with GPT-4 turbo completions:
  - No robots (~10k instructions): https://huggingface.co/datasets/BramVanroy/no_robots_dutch
  - UltraChat (~200k instructions): https://huggingface.co/datasets/BramVanroy/ultrachat_200k_dutch
  - UltraFeedback (DPO with GPT4+GEITje chat, ~50k): https://huggingface.co/datasets/BramVanroy/ultra_feedback_dutch
  - Orca DPO Pairs  (DPO with GPT4+GEITje chat, ~10k): https://huggingface.co/datasets/BramVanroy/orca_dpo_pairs_dutch

samusenps osanseviero dragonkue beomi p208p2002 gsarti s3nh taufiqdp victor Stopwolf Robbert jvdgoltz seostar clem cast42 jvh ajrogier

samusenps osanseviero gsarti HuggyMonkey


Introducing model-similarities, a new simple tool to contrast two models

A straightforward yet insightful tool designed to shed light on the similarities between various models. Discover it now at [Model Similarity GitHub Repository](https://github.com/fblgit/model-similarity).

This project is in its nascent stages, and we're eager for contributions and enhancements. Crafted with simplicity at its core, the tool performs two primary comparisons:
- Weight similarities, utilizing a simple approach to contrast vector differences (A != B).
- Cosine similarity between the parameters of models A and B, providing a nuanced measure of their alignment.

Included in the repository are sample analyses and reports that validate model card claims, particularly regarding the training specifics of transformer components such as MLP, Attention, etc. Remarkably, these samples reveal 100% similarity scores between those parts of the models, pinpointing the exact base model utilized.

Join us in refining and expanding this tool. Whether you're looking to contribute code, ideas, or both, your input will help transform this into a resource for everyone.

fblgit shivamanbhule thomasgauthier osanseviero distantquant victor samusenps shuvom mohammedbriman santiviquez clem alielfilali01


🙋🏻♂Hey there folks ,

i'm so impressed by the reception that https://huggingface.co/collabora/whisperspeech has recieved !

Check out the cool demo here :  https://huggingface.co/spaces/collabora/WhisperSpeech

- Open issue : how do we provide MPS support ? cc. @intel :-) looking into this now , any leads welcome!

check out also [collabora/whisperfusion](https://github.com/collabora/WhisperFusion)

hope you enjoy ! 🤗

blanchon samusenps matlok tollefj osanseviero victor clem


🚨📢🚀 Introducing Hercules-v2.0! A robust, multifaceted dataset for advanced models to excel in specialized domains. 🔬🌌📚🚀

📈 1.3M examples from sources derived from OpenHermes-2.5, covering Biology, Physics, Math, CS, Instruction Following, Function Calling, and Roleplay.

🔬 Enhance natural language understanding and processing in diverse domains.

🚀 Develop models for complex instructions, function calls, and roleplay scenarios.

📄 Licensed under Apache-2.0.

Thank you to all contributors and OpenHermes-2.5 creator! 🎉

Check it out here: https://huggingface.co/datasets/Locutusque/hercules-v2.0

📣 Update: After fine-tuning Mistral 7B on 100,000 examples of Hercules-v2.0, it earns an average score of 62 on Open LLM Leaderboard, outperforming OpenHermes-2.5 and OpenChat-3.5. 🎉

Check out this model here: https://huggingface.co/Locutusque/Hercules-2.0-Mistral-7B

taufiqdp osanseviero Tonic samusenps victor Locutusque kramp Severian clem bunnycore sethuiyer statler


GPU Poor POV: Burnout

Sometimes we do not have an energy to post about AI and new methods.
And thats totally ok, I guess.
Remember to sleep well and drink a lot of water. Have a great day :D <3

vicgalle Epiculous Ji-Ha samusenps visheratin MaziyarPanahi osanseviero joey00072 not-lain saishf victor Meggido PerceptiveFocusInc mvaloatto antiven0m taufiqdp kramp ajibawa-2023 award40 s3nh lusstta alielfilali01 Tonic mertbozkir KnutJaegersberg vincentweisser sbrandeis AtAndDev


ICYMI! Nomic Embed, the first fully open long context text embedder to beat OpenAI

- Open source, open weights, open data
- Beats OpenAI text-embeding-3-small and Ada on short and long context benchmarks
- Day 1 integrations with Langchain, LlamaIndex, MongoDB, and Sentence Transformers

Check out https://huggingface.co/nomic-ai/nomic-embed-text-v1 for the model weights.

Technical report: https://static.nomic.ai/reports/2024_Nomic_Embed_Text_Technical_Report.pdf
Blog Post: https://blog.nomic.ai/posts/nomic-embed-text-v1
Original Tweet Thread: https://x.com/nomic_ai/status/1753082063048040829?s=20

bstadt osanseviero davanstrien clem jeffboudier yjernite MaziyarPanahi julien-c SwimminPizza dave3991 samusenps baitao666 VictorSanh radames blanchon Rexhaif Tonic

osanseviero davanstrien clem yjernite julien-c VictorSanh radames


Lots of cool Gradio custom components, but is the most generally useful one I've seen so far: insert a Modal into any Gradio app by using the `modal` component!

```py
from gradio_modal import Modal

with gr.Blocks() as demo:
    gr.Markdown("### Main Page")
    gr.Textbox("lorem ipsum " * 1000, lines=10)

    with Modal(visible=True) as modal:
        gr.Markdown("# License Agreement")
```

victor s3nh sbarman25 eskayML akhaliq RSHVR dwipper

MaziyarPanahi afrideva samusenps akhaliq AseidasVera RSHVR


I'm happy to announce that ✨ Image to Music v2 ✨ is ready for you to try and i hope you'll like it too ! 😌

This new version has been crafted with transparency in mind,
so you can understand the process of translating an image to a musical equivalent.

How does it works under the hood ? 🤔

First, we get a very literal caption from https://huggingface.co/microsoft/kosmos-2-patch14-224; this caption is then given to a LLM Agent (currently https://huggingface.co/HuggingFaceH4/zephyr-7b-beta )which task is to translate the image caption to a musical and inspirational prompt for the next step.

Once we got a nice musical text from the LLM, we can send it to the text-to-music model of your choice:
MAGNet, MusicGen, AudioLDM-2, Riffusion or Mustango

Instead of the previous version of Image to Music which used Mubert API, and could output curious and obscure combinations, we only provide open sourced models available on the hub, called via the gradio API.

Also i guess the music result should be more accurate to the atmosphere of the image input, thanks to the LLM Agent step.

Pro tip, you can adjust the inspirational prompt to match your expectations, according to the chosen model and specific behavior of each one 👌

Try it, explore different models and tell me which one is your favorite 🤗
—› https://huggingface.co/spaces/fffiloni/image-to-music-v2

victor samusenps osanseviero phanes abidlabs koheingt adhisetiawan Indomitable-ai boqsc deepkyu mvaloatto MaziyarPanahi powzone kelsy Chief-Inspector pe65374 Swhalen98 aceeee Aixbox

naorex ivied7 MehdiLeZ eisneim OmbelineM


First post alert 🚨

Super excited to share with you my first Chat assistant :

`HuggingAssist`, meant to offer guidance with the large HuggingFace ecosystem

Chat with it from here : https://hf.co/chat/assistant/65bd0adc08560e58be454d86

It would be more helpful when the RAG / WEB features are available !
Looking forward to it 🔥

ps : tnx @Chunte for the cool Huggies

osanseviero nsarrazin victor EmilyWitko clem pierrci abidlabs samusenps BrigitteTousi KBayoud antmaiorino Tohito kramp mammour


ReAGent: Towards A Model-agnostic Feature Attribution Method for
  Generative Language Models (2402.00794)

🔍 Today's pick in Interpretability & Analysis of LMs: ReAGent: Towards A Model-agnostic Feature Attribution Method for Generative Language Models  by @casszhao and B. Shan

Authors propose Recursive Attribution Generation (ReAGent), a perturbation-based feature attribution approach specifically conceived for generative LMs. The method employs a lightweight encoder LM to replace sampled input spans with valid alternatives and measure the effect of the perturbation on the drop in next token probability predictions. ReAGent is shown to consistentlyoutperform other established approaches across several models and generation tasks in terms of token- and sentence-level faithfulness.

📄 Paper: https://huggingface.co/papers/2402.00794
💻 Code: https://github.com/casszhao/ReAGent


🔥 New LLM leaderboard on the hub: NPHardEval!

It uses questions of logic, of different mathematical complexities, as a proxy for reasoning abilities. It notably removes questions relying on arithmetic, to really focus on logical abilities.
What's interesting imo is the potential to really study a model performance at different levels of complexity.

Bonus: Since the questions can be generated automatically, it's going to be dynamic, updated monthly! 🚀
https://huggingface.co/spaces/NPHardEval/NPHardEval-leaderboard

Read more about how their questions are generated in the intro blog: https://huggingface.co/blog/leaderboards-on-the-hub-nphardeval

Congrats to @lizhouf, @wenyueH, @hyfrankl and their teams!

alvarobartt osanseviero alielfilali01 samusenps lizhouf diwank hyfrankl kramp joaogante hivaze Kukedlc sbrandeis


OLMo: Accelerating the Science of Language Models (2402.00838)

OLMo

Accelerating the Science of Language Models

paper page: https://huggingface.co/papers/2402.00838

a state-of-the-art, truly Open Language Model and its framework to build and study the science of language modeling. Unlike most prior efforts that have only released model weights and inference code, we release OLMo and the whole framework, including training data and training and evaluation code. We hope this release will empower and strengthen the open research community and inspire a new wave of innovation.

vladbogo Dlbk osanseviero samusenps clem taufiqdp mathiasn1 blanchon matlok Ji-Ha Iliassti nold Keesh SX-GitHub


🔥 New on HuggingChat: Assistants!

Today we are releasing Assistants on HuggingChat!
Assistants are a fun way to package your prompts and share them with the world - powered by Open source Models of course!

Learn more about Assistants here: https://huggingface.co/spaces/huggingchat/chat-ui/discussions/357
Browse Assistants here: https://huggingface.co/chat/assistants

hmb dtripathi Dlbk Sylvestre AmethystAmenity yasserrmd osanseviero andrewrreed chkla alielfilali01 kramp samusenps clem taufiqdp adamelliotfields blanchon beomi marianbasti Nymbo

PierreLepagnol samusenps neovalle clem fatemex blanchon beomi Nymbo


If you're looking for geospatial datasets, you might find what you're looking for in the Geospatial Datasets Collections:
https://huggingface.co/collections/blanchon/geospatial-datasets-656ddb097c934a7b3c4d4619

s3nh osanseviero clem samusenps doof-ferb severo


GPU Poor POV: Quantization

Today I want to share with you my notebook plug and play code
which help me a lot through my quantization journey.
Hope youll find it interesting it could be a good starter point to
gguf some of your awesome models :)

Have a great day <3

https://s3nh.bearblog.dev/gpu-poor-pov-gguf-snippet/

lunarflu Aspik101 clem naveedtrainer osanseviero samusenps Csplk taufiqdp neovalle Tanvir1337 Ji-Ha jtatman AtAndDev

lunarflu clem naveedtrainer impactframes Tanvir1337 alielfilali01 s3nh Ji-Ha AtAndDev

lunarflu clem naveedtrainer osanseviero Tanvir1337 Ji-Ha AtAndDev


Just out: new custom Gradio component specifically designed for code completion models 🔥

hmb dhuynh95 sbarman25 julien-c osanseviero victor lunarflu clem yehiaserag samusenps aust-t alielfilali01 gsarti ThreadAbort not-lain calmgoose deepkyu freddyaboulton


release day release day! OLMo 1b + 7b out today 🥳 and 65b coming soon...

With OLMo, we are really focused on advancing the study of LLMs. We release **everything**, from toolkit to create its training dataset (dolma) to training & inference code:

- OLMo paper: https://allenai.org/olmo/olmo-paper.pdf
- OLMo train code: https://github.com/allenai/OLMo
- OLMo eval code: https://github.com/allenai/OLMo-Eval
- OLMo 7b: https://huggingface.co/allenai/OLMo-7B
- OLMo 1b: https://huggingface.co/allenai/OLMo-1B
- Dolma paper: https://allenai.org/olmo/dolma-paper.pdf
- Dolma dataset v1.6: https://huggingface.co/datasets/allenai/dolma
- Dolma toolkit v1.0:  https://github.com/allenai/dolma

osanseviero dim julien-c andysalerno juancopi81 lunarflu clem davanstrien mdouglas samusenps kramp yizhongw beomi alielfilali01 mathiasn1 adamelliotfields vbuhoijymzoi Tonioesparza loubnabnl dashfunnydashdash


Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion
  Tokens (2401.17377)

Infini-gram

Scaling Unbounded n-gram Language Models to a Trillion Tokens

demo: https://huggingface.co/spaces/liujch1998/infini-gram

paper page: https://huggingface.co/papers/2401.17377

train them at the same data scale as neural LLMs -- 1.4 trillion tokens. This is the largest n-gram model ever built. Second, existing n-gram models use small n which hinders their performance; we instead allow n to be arbitrarily large, by introducing a new infty-gram LM with backoff. Instead of pre-computing n-gram count tables (which would be very expensive), we develop an engine named infini-gram -- powered by suffix arrays -- that can compute infty-gram (as well as n-gram with arbitrary n) probabilities with millisecond-level latency. The infty-gram framework and infini-gram engine enable us to conduct many novel and interesting analyses of human-written and machine-generated text: we find that the infty-gram LM has fairly high accuracy for next-token prediction (47%), and can complement neural LLMs to greatly reduce their language modeling perplexities. When analyzing machine-generated text, we also observe irregularities in the machine--infty-gram agreement level with respect to the suffix length, which indicates deficiencies in neural LLM pretraining and the positional embeddings of Transformers. We open-source our infini-gram engine in the hopes of enabling more study on how to best use verbatim information retrieved from large text corpora.

sankeerth1729 Maani lunarflu clem osanseviero soldni sacharbit matlok


Vision LLM for #edgecomputing ?

@openbmb, who OS'ed the UltraFeedback dataset before, released a series of strong eco-friendly yet powerful LLMs

- MiniCPM: 2B model that competes with Mistral-7B

- MiniCPM-V: 3B vision LLM on edge!
- MiniCPM-V: 3B vision LLM on edge!


Good folks at @Apple have developed a novel method called KV Prediction that significantly reduces the "time to first token" (TTFT) for on-device LLM inference.

Some highlights of the paper:

• Uses a small auxiliary transformer model to efficiently predict the KV cache of a larger base model
• Reduces TTFT by up to 4x while retaining 60-80% accuracy on benchmarks
• Achieves Pareto-optimal efficiency-accuracy trade-off compared to baselines
• Demonstrates 15-50% relative accuracy improvements on TriviaQA at equal TTFT FLOP budgets
• Shows up to 30% accuracy gains on HumanEval code completion at fixed TTFT FLOP counts
• Validated on Apple M2 Pro CPU, proving FLOP gains translate to real-world speedups

So, how's it done?

Based on the KV Prediction method described in the paper, here are the key steps for how it's done:

1. Choose a base model and an auxiliary model:
- The base model is a larger, pretrained transformer model that will be used for final generation.
- The auxiliary model is a smaller transformer model used to efficiently process the input prompt.

2. Design the KV predictor:
- Create a set of learned linear projections to map from the auxiliary model's KV cache to the base model's KV cache.
- Define a mapping from auxiliary cache layers to base cache layers.

3. Training process:
- Pass input tokens through the auxiliary model to get its KV cache.
- Use the KV predictor to generate a predicted KV cache for the base model.
- Run the base model using the predicted KV cache and compute losses.
- Backpropagate errors through the frozen base model to update the auxiliary model and KV predictor.

4. Inference process:
- Process the input prompt with the auxiliary model to get its KV cache.
- Use the KV predictor to generate the predicted base model KV cache.
- Run a single token generation step with the base model using the predicted KV cache.
- Continue autoregressive generation with the base model as normal.

Excited to hear your thoughts!

John6666 YaTharThShaRma999 barakplasma KingNish myronkoch AtAndDev GoDjMike


A new `timm` release (1.0.11) is out now. A also wrote an article on one of the included models: https://huggingface.co/blog/rwightman/mambaout

Featured in the release are:
  * The MambaOut model, a cheeky arch inspired by SSM but without the SSM part, a ConvNeXt with gating.
  * Several timm trained MambaOut variations with arch tweaks and ImageNet-12k pretrain to verify scaling, supplement ported weights.
  * The smallest MobileNetV4, a 0.5x width scaled Conv-Small.
  * Two impressive MobileNetV3 Large models outperforming all previous, using MNV4 Small recipe.
  * 'Zepto,' a new compact ConvNeXt variant even smaller than the previous Atto, 2.2M params, RMSNorm, and solid results for its size.
  * Newly ported SigLIP SO400M/16 ViT multi-lingual weights, the largest i18n weights, prevous was B/16.
  * Two ImageNet-1k fine-tuned SigLIP SO400M models at 378x378
  * InternViT 300M weight port. A really solid ViT encoder distilled from OpenGVLab 6B VL model encoder.
  * An assortment of very small, sub 1M param pretrained test models to improve library unit tests and serve low-resource applications.


China is advancing rapidly in AI technology while maintaining a strong focus on governance 🇨🇳📑
We've collected key AI governance documents released since 2017 and will continue updating them in this organization on the hub 👉China LLMs on Hugging Face
✨ https://huggingface.co/spaces/zh-ai-community/china-ai-policy-research
Any feedback is welcome🤗


What a great day for Open Science! @AIatMeta released models, datasets, and code for many of its research artefacts! 🔥

1. Meta Segment Anything Model 2.1: An updated checkpoint with improved results on visually similar objects, small objects and occlusion handling. A new developer suite will be added to make it easier for developers to build with SAM 2.

Model checkpoints: https://huggingface.co/collections/reach-vb/sam-21-6702d40defe7611a8bafa881

2. Layer Skip: Inference code and fine-tuned checkpoints demonstrating a new method for enhancing LLM performance.

Model checkpoints: https://huggingface.co/collections/facebook/layerskip-666b25c50c8ae90e1965727a

3. SALSA: New code enables researchers to benchmark AI-based attacks to validate security for post-quantum cryptography.

Repo: https://github.com/facebookresearch/LWE-benchmarking

4. Meta Lingua: A lightweight and self-contained codebase designed to train language models at scale.

Repo: https://github.com/facebookresearch/lingua

5. Meta Open Materials: New open source models and the largest dataset to accelerate AI-driven discovery of new inorganic materials.

Model checkpoints: https://huggingface.co/fairchem/OMAT24

6. MEXMA: A new research paper and code for our novel pre-trained cross-lingual sentence encoder covering 80 languages.

Model checkpoint: https://huggingface.co/facebook/MEXMA

7. Self-Taught Evaluator: a new method for generating synthetic preference data to train reward models without relying on human annotations.

Model checkpoint: https://huggingface.co/facebook/Self-taught-evaluator-llama3.1-70B

8. Meta Spirit LM: An open-source language model for seamless speech and text integration.

Repo: https://github.com/facebookresearch/spiritlm

abidlabs John6666 pcuenq clem blanchon nbroad kuotient


🕵♂ Let’s investigate NVIDIA's Llama-3.1-Nemotron-70B-Instruct-HF model performance - great analysis by @alozowski!

https://x.com/ailozovskaya/status/1847278827925291370


📈 Increase the quality of your RAG with a simple Linear Layer! No need to change your embedding model (keep that old OpenAI API).

Introducing EmbeddingAlign RAG, a novel approach to improve Retrieval-Augmented Generation (RAG) systems.

Key highlights:
- Uses a simple linear transformation on existing embeddings
- Boosts hit rate from 89% to 95% on real-world examples
- Minor increase on latency (less than 10ms)
- Works on top of blackbox embedding models (Mistral AI, OpenAI, Cohere,...)
- No dataset needed (just your documents)
- Train easily on CPU

🤗 Read the full article here on HF: https://huggingface.co/blog/PLB/embedding-align-rag


𝟮. 𝗟𝗮𝘁𝗲-𝗶𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝗼𝗻: 𝘁𝗵𝗶𝘀 𝗶𝘀 𝗖𝗼𝗹𝗕𝗘𝗥𝗧 🏃

𝟮. 𝗟𝗮𝘁𝗲-𝗶𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝗼𝗻: 𝘁𝗵𝗶𝘀 𝗶𝘀 𝗖𝗼𝗹𝗕𝗘𝗥𝗧 🏃

𝟯. 𝗘𝗮𝗿𝗹𝘆 𝗶𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝗼𝗻: 𝗖𝗿𝗼𝘀𝘀-𝗲𝗻𝗰𝗼𝗱𝗲𝗿𝘀 🏋

𝟯. 𝗘𝗮𝗿𝗹𝘆 𝗶𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝗼𝗻: 𝗖𝗿𝗼𝘀𝘀-𝗲𝗻𝗰𝗼𝗱𝗲𝗿𝘀 🏋

𝗛𝗼𝘄 𝘁𝗼 𝗿𝗲-𝗿𝗮𝗻𝗸 𝘆𝗼𝘂𝗿 𝘀𝗻𝗶𝗽𝗽𝗲𝘁𝘀 𝗶𝗻 𝗥𝗔𝗚 ⇒ ColBERT, Rerankers, Cross-Encoders

Let’s say you’re doing RAG, and in an effort to improve performance, you try to rerank a few possible source snippets by their relevancy to a query.

How can you score similarity between your query and any source document? 🤔 📄 ↔ 📑

𝟭. 𝗝𝘂𝘀𝘁 𝘂𝘀𝗲 𝗲𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴𝘀 : 𝗡𝗼-𝗶𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝗼𝗻 🏎

This means that you encode each token from both the query and the doc as separate vectors, then average the tokens of each separately to get in total 2 vectors, then you compute similarity via cosine or something.
➡ Notable examples: Check the top of the MTEB leaderboard!

𝟮. 𝗟𝗮𝘁𝗲-𝗶𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝗼𝗻: 𝘁𝗵𝗶𝘀 𝗶𝘀 𝗖𝗼𝗹𝗕𝗘𝗥𝗧 🏃

These encode each token from both query and doc as separate vectors as before, but compare all together without previously averaging them and losing information.

This is more accurate than no-interaction but also slower because you have to compare n*m vectors instead of 2. At least you can store documents in memory. And ColBERT has some optimisations like pooling to be faster.

➡ Notable examples: ColBERTv2, mxbai-colbert-large-v1, jina-colbert-v2

𝟯. 𝗘𝗮𝗿𝗹𝘆 𝗶𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝗼𝗻: 𝗖𝗿𝗼𝘀𝘀-𝗲𝗻𝗰𝗼𝗱𝗲𝗿𝘀 🏋

Basically you run the concatenated query + document in a model to get a final score.

This is the most accurate, but also the slowest since it gets really long when you have many docs to rerank! And you cannot pre-store embeddings.

➡ Notable examples: MixedBread or Jina AI rerankers!

🚀 So what you choose is a trade-off between speed and accuracy: I think ColBERT is often a really good choice!

Based on this great post by Jina AI 👉 https://jina.ai/news/what-is-colbert-and-late-interaction-and-why-they-matter


Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (2410.02073)

It's raining depth estimation models ☔
DepthPro is a zero-shot depth estimation model by Apple, it's fast, sharp and accurate 🔥
Demo: https://huggingface.co/spaces/akhaliq/depth-pro
Model: https://huggingface.co/apple/DepthPro
Paper page: https://huggingface.co/papers/2410.02073

The model consists of two encoders: an encoder for patches and an image encoder 🖼 The outputs of both are merged to decode to depth maps and get the focal length.
The model outperforms the previous state-of-the-art models in average of various benchmarks 📑

victor YaTharThShaRma999 John6666 baratpaim adorkin mpottinger AtAndDev


✨ 𝗧𝗼𝗽 𝗛𝗶𝗴𝗵𝗹𝗶𝗴𝗵𝘁𝘀:

✨ 𝗧𝗼𝗽 𝗛𝗶𝗴𝗵𝗹𝗶𝗴𝗵𝘁𝘀:

📣 𝚑𝚞𝚐𝚐𝚒𝚗𝚐𝚏𝚊𝚌𝚎_𝚑𝚞𝚋 v0.26.0 is out with some new features and improvements!

✨ 𝗧𝗼𝗽 𝗛𝗶𝗴𝗵𝗹𝗶𝗴𝗵𝘁𝘀:
- 🔐 Multiple access tokens support: Easily manage multiple access tokens with new CLI commands. Perfect for handling multiple tokens with specific permissions in production or when collaborating with external teams.
- 🖼 Conversational VLMs inference is now supported with InferenceClient's chat completion!
- 📄 Daily Papers API: Seamlessly search and retrieve detailed paper information from the Hub!

We’ve also introduced multiple bug fixes and quality-of-life improvements - thanks to the awesome contributions from our community! 🤗

Check out the release notes here: https://huggingface.co/spaces/Wauplin/huggingface_hub/discussions/9

and you can try it out now 👇
```
pip install huggingface_hub==0.26.0
```

Chris4K victor asoria John6666 AtAndDev Nymbo louisbrulenaudet


H2O-Danube-1.8B Technical Report (2401.16818)

Happy to share H2O-Danube-1.8b, a small 1.8b model based trained on only 1T natural language tokens showing competitive metrics across benchmarks in the <2B model space.

Base weights: https://huggingface.co/h2oai/h2o-danube-1.8b-base
Chat weights: https://huggingface.co/h2oai/h2o-danube-1.8b-chat
Technical report: https://huggingface.co/papers/2401.16818

osanseviero Maani lunarflu mdouglas stefan-jo


It seems February started with a fully open source AI renaissance 🌟

Models released with fully open dataset, training code, weights ✅

LLM - https://huggingface.co/collections/allenai/olmo-suite-65aeaae8fe5b6b2122b46778 🧠
Embedding - https://huggingface.co/nomic-ai/nomic-embed-text-v1 📚 (sota!)

And it's literally February 1st - can't wait to see what else the community will bring 👀

clem multimodalart osanseviero BrigitteTousi Maani lunarflu leigh301 dillfrescott zamply alielfilali01 impactframes MdnghtToker OmbelineM

fffiloni multimodalart BrigitteTousi lunarflu clem greencookies dillfrescott


Today, we’re releasing our first pretrained Open Language Models (OLMo) at the Allen Institute for AI (AI2), a set of 7 billion parameter models and one 1 billion parameter variant. This line of work was probably the main reason I joined AI2 and is the biggest lever I see possible to enact meaningful change in how AI is used, studied, and discussed in the short term.

Links at the top because that's what you want:
* Core 7B model: https://huggingface.co/allenai/OLMo-7B
* 7B model twin (different GPU hardware): https://huggingface.co/allenai/OLMo-7B-Twin-2T
* 1B model: https://huggingface.co/allenai/OLMo-1B
* Dataset: https://huggingface.co/datasets/allenai/dolma
* Paper (arxiv soon): https://allenai.org/olmo/olmo-paper.pdf
* My personal blog post: https://www.interconnects.ai/p/olmo

OLMo will represent a new type of LLM enabling new approaches to ML research and deployment, because on a key axis of openness, OLMo represents something entirely different. OLMo is built for scientists to be able to develop research directions at every point in the development process and execute on them, which was previously not available due to incomplete information and tools.

Depending on the evaluation methods, OLMo 1 is either the best 7 billion parameter base model available for download or one of the best. This relies on a new way of thinking where models are judged on parameter plus token budget, similar to how scaling laws are measured for LLMs.

We're just getting started, so please help us learn how to be more scientific with LLMs!

julien-c osanseviero thomwolf AdinaY clem DnzzL LZDXN PereLluis13 venkateshmurugadas 90Barricade93 linoyts rbiswasfc Hamish victor yleo BrigitteTousi ArunkumarVR samforeman Maani lunarflu davanstrien socialprescribing kramp beomi alielfilali01 jeffboudier Tanvir1337 patrickocal Gilojond thewise eswardivi anileo1 shtoshni ahkarami radames

thomwolf clem fffiloni victor BrigitteTousi lunarflu socialprescribing beomi ahkarami radames

osanseviero clem victor BrigitteTousi lunarflu beomi alielfilali01


Had a lot of fun making this plot today.

If someone ever asks you why you need ML monitoring, show them this picture 😂


In-Context Language Learning: Architectures and Algorithms (2401.12973)

🔍 Today's pick in Interpretability & Analysis of LMs: Gradient-Based Language Model Red Teaming  by N. Wichers, C. Denison and @beirami

This work proposes Gradient-Based Red Teaming (GBRT), a red teaming method for automatically generating diverse prompts inducing an LM to output unsafe responses.

In practice, prompts are learned by scoring LM responses with a safety-trained probing classifier, and back-propagating through frozen classifier and LM to update the prompt.

Authors experiment with variants of GBRT aimed at inducing realistic prompts in an efficient way, and GBRT prompts are more likely to generate unsafe responses than those found by established RL-based red teaming methods. Moreover, these attacks are shown to succeed even when the LM has been fine-tuned to produce safer outputs.

📄 Paper: https://huggingface.co/papers/2401.12973
💻 Code: https://github.com/google-research/google-research/tree/master/gbrt

clem osanseviero lunarflu beirami 9voltfan2009


Let me introduce you LLE:  Leaks, leaks everywhere!

A quick experiment I've carried out on around 600 datasets from the HF Hub, the results are stored in https://huggingface.co/datasets/lbourdois/LLE, and the methodology is described in
https://huggingface.co/blog/lbourdois/lle

tomaarsen Dlbk vuluu clem osanseviero dhuynh95 alielfilali01 gsarti


Introducing Gajendra!

An early release of our 7B Hindi-Hinglish-English Instruction fine-tuned language model.

Model: https://huggingface.co/BhabhaAI/Gajendra-v0.1

We additionally explore ways to filter examples that can be translated from English to Hindi and are releasing initial versions of both dataset and model for it.

Model: https://huggingface.co/BhabhaAI/Mistral-translation-classify
Dataset: https://huggingface.co/datasets/BhabhaAI/translation-classify

Looking forward to collaborate with open source community to accelerate and release Hindi LLMs.

samusenps VanshGehlot victor osanseviero julien-c sbarman25 satpalsr alielfilali01 smangrul


What is WhisperKit?
WhisperKit is a pure Swift package built by @argmaxinc  that enables real-time Whisper inference on iPhone, Mac and even Apple Watch in two lines of code. It is built on Core ML and can be configured to run on the Neural Engine or GPU or both. WhisperKit models are published and distributed through the Hugging Face Hub. Anyone can publish a Whisper fine-tune and evaluate it on their own dataset using WhisperKit!

WhisperKit models: https://huggingface.co/argmaxinc/whisperkit-coreml
Example App: https://testflight.apple.com/join/LPVOyJZW
Swift Package: https://github.com/argmaxinc/WhisperKit
Python toolkit: https://github.com/argmaxinc/whisperkittools

osanseviero victor VanshGehlot C00100011 fffiloni ZachNagengast dillfrescott Araeynn

samusenps vishalvis ZachNagengast dillfrescott Araeynn pcuenq


Introducing the "UltraTextbooks" dataset 🚀📚
Check it out here: https://huggingface.co/datasets/Locutusque/UltraTextbooks
📘 A comprehensive collection of high-quality synthetic and human-written textbooks
👨🎓 Spanning various subjects and programming languages
🔧 Designed for advanced NLP tasks like language modeling, educational QA, text summarization, and content generation for edu purposes
🚀 Future expansions planned with additional data sources to enhance the corpus
👇 Data composition highlights 👇
- Blend of synthetic and human-written material
- Includes topics from general edu to specialized areas
- Structured with field "text"
🧩 Data collection from various Hugging Face datasets, guided by a diverse and comprehensive curation rationale
🚧 Limitations may exist, so report any issues you encounter

samusenps Felladrin osanseviero vicgalle VanshGehlot dhuynh95 Locutusque aloobun afrideva Lee1990

pszemraj Pelmeshek osanseviero Locutusque qinluo Lee1990


moondream1 can now be used directly from transformers!

Felladrin osanseviero clem cnmoro maywell samusenps Locutusque VanshGehlot fffiloni not-lain Sylvestre levisz lrzjason ajibawa-2023 Aptronym

YaTharThShaRma999 sujitvasanth osanseviero talgaurdian clem taufiqdp not-lain piebro impactframes Nymbo


📣 NEW on HF

the Dataset Viewer is now available on *private datasets* too

You need to be a PRO or a Enterprise Hub user. 🔥

Great work from our Datasets team 🥰: @lhoestq @severo @polinaeterna @asoria @albertvillanova and the whole team 🥰

lhoestq severo clem blanchon osanseviero pierrci albertvillanova VanshGehlot vladbogo AdinaY BrigitteTousi michaljunczyk not-lain Nymbo

fffiloni clem osanseviero Pelmeshek kramp mvaloatto awacke1 AdinaY BrigitteTousi Tonic not-lain Nymbo


🔥 New LLM leaderboard on the hub: an Enterprise Scenarios Leaderboard!

This work evaluates LLMs on several real world use cases (Finance documents, Legal confidentiality, Customer support, ...), which makes it grounded, and interesting  for companies! 🏢
Bonus: the test set is private, so it's hard to game 🔥
https://huggingface.co/spaces/PatronusAI/enterprise_scenarios_leaderboard

Side note: I discovered through this benchmark that you could evaluate "Engagingness" of an LLM, which could also be interesting for our LLM fine-tuning community out there.

Read more about their different tasks and metrics in the intro blog: https://huggingface.co/blog/leaderboards-on-the-hub-patronus

Congrats to @sunitha98 who led the leaderboard implementation, and to @rebeccaqian and @anandnk24, all at Patronus AI !


InstantID-2V is out ! ✨

It's like InstantID, but you get a video instead. Nothing crazy here, it's simply a shortcut between two demos.

Let's see how it does work with gradio API:

1. We call https://huggingface.co/spaces/InstantX/InstantID with a conditional pose from cinematic camera shot (example provided in the demo)
2. Then we send the previous generated image to https://huggingface.co/ali-vilab/i2vgen-xl

Et voilà 🤗 Try it : https://huggingface.co/spaces/fffiloni/InstantID-2V

—
Note that generation can be quite long, so take the opportunity to brew you some coffee 😌
If you want to skip the queue, you can of course reproduce this pipeline manually

linoyts akhaliq yang100 phanes PeepDaSlan9 samusenps victor osanseviero clem echobingo147 AdinaY cansa danielus


🚀 The Open Source AI community needs more open datasets for improving Open LLMs. Excited to share our new open dataset for boosting chat models:

🎉 Welcome Distilabel Capybara DPO, a multi-turn, high-quality preference dataset.

https://huggingface.co/datasets/argilla/distilabel-capybara-dpo-7k-binarized

Why?
Best closed chat models are built on top of multi-turn dialogue preference data. The OSS community lacks these datasets. This dataset is the first in the series to close this gap.

Is this dataset useful?
To test this dataset, we've built our virtual launching partner:

🎉 Welcome CapybaraHermes, a preference tuned OpenHermes with increased second turn capabilities on MTBench

https://huggingface.co/argilla/CapybaraHermes-2.5-Mistral-7B

As usual, models are the least important to us. We like to focus on the data. Our mission is to build and share high-quality datasets, sharing our methods in the open so the community can improve upon them.

That's why, we took some time to describe the full methodology on the dataset card, check it out and give us feedback! Data and methods are never perfect!

Finally, this is just a preview version and would love to collaborate with you to add more benchmarking results, what hyperparams work for DPO'ing models, what mix of datasets, etc.

Expect some more datasets in the coming weeks. Let's build the best data for AI, together.

davanstrien lhoestq osanseviero victor severo julien-c samusenps clem Smorty100 Arjun4707 gabrielmbmb kramp mayurninama davidberenstein1957 hfthorvaldur leonardlin alvarobartt alielfilali01 sbrandeis


Weaver: Foundation Models for Creative Writing (2401.17268)

Weaver: Foundation Models for Creative Writing

https://huggingface.co/papers/2401.17268

most-capable Weaver Ultra model surpasses GPT-4, a state-of-the-art generalist LLM, on various writing scenarios, demonstrating the advantage of training specialized LLMs for writing purposes.


Explaining a new state-of-the-art monocular depth estimation model: Depth Anything ✨ 🧶
Before we begin: Depth Anything is recently integrated to 🤗 transformers and you can use it with three lines of code! ✨
```python
from transformers import pipeline

pipe = pipeline(task="depth-estimation", model="LiheYoung/depth-anything-small-hf")
depth = pipe(image)["depth"]
```
We have also built an app for you to compare different depth estimation models 🐝 🌸  https://huggingface.co/spaces/merve/compare_depth_models
Check out Depth Anything in Web by @Xenova https://huggingface.co/spaces/Xenova/depth-anything-web

The model's success heavily depends on unlocking the use of unlabeled datasets, although initially the authors used self-training and failed.
What the authors have done:
➰ Train a teacher model on labelled dataset
➰ Guide the student using teacher and also use unlabelled datasets pseudolabelled by the teacher
However, this was the cause of the failure, as both architectures were similar, the outputs were the same.
So the authors have added a more difficult optimization target for student to learn additional knowledge on unlabeled images that went through color jittering, distortions, Gaussian blurring and spatial distortion, so it can learn more invariant representations from them.
The architecture consists of DINOv2 encoder to extract the features followed by DPT decoder. At first, they train the teacher model on labelled images, and then they jointly train the student model and add in the dataset pseudo-labelled by ViT-L.
Thanks to this, Depth Anything performs very well! I have also benchmarked the inference duration of the model against different models here. I also ran torch.compile benchmarks across them and got nice speed-ups 🚀 https://huggingface2.notion.site/DPT-Benchmarks-1e516b0ba193460e865c47b3a5681efb?pvs=4

osanseviero clem Xenova Ar4ikov tusk3d marooz


TIL: https://huggingface.co/datasets/EleutherAI/pile is on Wikipedia: https://en.wikipedia.org/wiki/The_Pile_(dataset)


🔍 Today's pick in Interpretability & Analysis of LMs: In-Context Language Learning: Architectures and Algorithms by E. Akyürek, B. Wang, Y. Kim, J. Andreas

This work methodically evaluates of in-context learning on formal languages across several model architectures, showing that Transformers outperform all other recurrent and convolutional models, including SSMs. These results are attributed to the presence of “n-gram heads” able to retrieve
the token following a context already seen in the current context window and copy it. This idea is further supported by a better ability of Transformer models to encode in-context n-gram frequencies for n>1, and a higher similarity of Transformer-based LM outputs with classical n-gram models trained on the same data. Finally, these insights are applied to the design static attention layers mimicking the behavior of n-gram head, which lead to lower perplexity despite the lower computational costs.

📄 Paper: https://huggingface.co/papers/2401.12973
💻 Code: https://github.com/berlino/seq_icl


Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation
  in Natural Language Generation (2302.09664)

Pretty novel idea on how to estimate *semantic* uncertainty. 🤔

Text generation tasks are challenging because a sentence can be written in multiple ways but still preserve its meaning.

For instance, "France's capital is Paris" means the same as "Paris is France's capital." 🇫🇷

In uncertainty quantification, we often look at token-level probabilities to quantify how "confident" an LLM is about its output. However, in this paper, the authors look at uncertainty at a meaning level.

Their motivation is that meanings are especially important for LLMs' trustworthiness; a system can be reliable even with many different ways to say the same thing, but answering with inconsistent meanings shows poor reliability.

To estimate semantic uncertainty, they introduce an algorithm for clustering sequences that mean the same thing, based on the principle that two sentences mean the same thing if we can infer one from the other. 🔄🤝

Then, they determine the likelihood of each meaning and estimate the semantic entropy by summing probabilities that share a meaning.

There's a lot more to it, but their results look quite nice when compared with non-semantic approaches.

Paper: https://huggingface.co/papers/2302.09664

osanseviero victor gsarti merve clem Citaman


The next version of Gradio will be significantly more efficient (as well as a bit faster) for anyone who uses Gradio's streaming features. Looking at you chatbot developers @oobabooga @pseudotensor :)

The major change that we're making is that when you stream data, Gradio used to send the entire payload at each token. This is generally the most robust way to ensure all the data is correctly transmitted. We've now switched to sending "diffs" --> so at each time step, we automatically compute the diff between the most recent updates and then only send the latest token (or whatever the diff may be). Coupled with the fact that we are now using SSE, which is a more robust communication protocol than WS (SSE will resend packets if there's any drops), we should have the best of both worlds: efficient *and* robust streaming.

Very cool stuff @aliabid94! PR: https://github.com/gradio-app/gradio/pull/7102

abbasm2 osanseviero yjernite oobabooga samusenps jeffboudier gsarti sam-kap satpalsr akhaliq alielfilali01 joaogante


"The Case for Co-Designing Model Architectures with Hardware"

This is a long overdue paper that we have started discussing back when training BLOOM-176.

Basically this paper tells you how to design your model's dimensions for an optimal training throughput.

Fantastic!

Yours truly contributed the SwiGLU section ;)

https://twitter.com/QuentinAnthon15/status/1752393989813375119

https://arxiv.org/abs/2401.14489

clem qanthony radames osanseviero abidlabs samusenps jeffboudier efederici merve muhtasham

clem osanseviero abidlabs merve muhtasham


So impressed with the speed and accuracy of https://huggingface.co/spaces/vikhyatk/moondream1 by
@vikhyatk (especially the last answer 😝😝😝).

Open multi-modal models have gone a long way!

Model: https://huggingface.co/vikhyatk/moondream1

vikhyatk victor osanseviero clem GloriaMK fffiloni akhaliq karankatke masonchambersiii abidlabs samusenps floattingabby gate369 jeffboudier ArthurZ combox snarktank michaelkleyn ivanfioravanti


Delighted to share a course I've learned a ton from about getting better outputs from LLMs

https://www.wandb.courses/courses/steering-language-models

We released it last Thursday (free) and at just 30 minutes of content total, its very information-dense with non-stop learnings covering important concepts around LLM validation, making your approach to LLM prompting more pythonic and quickly covers a basic RAG application at the end.

Would love to hear what ye think!

clem pierrci osanseviero altryne param-bharat merve mrdbourke kurianbenoy dillfrescott

clem osanseviero param-bharat merve dillfrescott


Quick build of the day: LCM Supa Fast Image Variation
—
We take the opportunity to combine moondream1 vision and LCM SDXL fast abilities to generate a variation from the subject of the image input.
All that thanks to gradio APIs 🤗

Try the space: https://huggingface.co/spaces/fffiloni/lcm-img-variations

samusenps victor clem osanseviero designfailure DamarJati merve


InternLM-XComposer2: Mastering Free-form Text-Image Composition and
  Comprehension in Vision-Language Large Model (2401.16420)

InternLM-XComposer2

Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model

paper page: https://huggingface.co/papers/2401.16420

Experimental results demonstrate the superiority of InternLM-XComposer2 based on InternLM2-7B in producing high-quality long-text multi-modal content and its exceptional vision-language understanding performance across various benchmarks, where it not only significantly outperforms existing multimodal models but also matches or even surpasses GPT-4V and Gemini Pro in certain assessments.