Black-Box Access is Insufficient for Rigorous AI Audits (2401.14446)

🔍 Today's pick in Interpretability & Analysis of LMs: Black-Box Access is Insufficient for Rigorous AI Audits by @stecas @carsonezell et al.

Audits conducted on AI systems can identify potential risks and ensure their compliance to safety requirements. Authors categorise audits based on the access to model-related resources (black, grey, white and out-of-the box) and highlight how levels of transparency on audited AI system enable broader and more effective auditing procedures. Technical, physical, and legal safeguards for performing audits are also introduced to ensure minimal security risks for audited companies. Authors conclude that transparency on the type of auditors’ access and methods is a pre-requisite to correctly interpret audit results, and white- and outside-the-box access allow for substantially more scrutiny than black-box access alone.

📄 Paper: https://huggingface.co/papers/2401.14446

🔍 Further readings:

📄Taxonomy of AI system access: https://bit.ly/struct-access
💻An API for transparent science on Black-box AI (NNsight): https://nnsight.net/about


EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models (2312.06281)

The most interesting LLM benchmark I've seen so far... reminder that there's lots of characterization of LLMs still yet to do.

https://huggingface.co/papers/2312.06281

bradwray wuyinansooymilk osanseviero clem doraemom0812


Fascinating paper by Rand shows that there is no statistically significant difference between using LLMs or regular internet to craft operational plans for bioweapons!

This is the first paper that actually studies the impact of AI on bioweapons from an operational perspective and looks at the big question: is AI any better than just using public data on the Internet?

As most of the data is most likely out there, an LLM would just be a more efficient tool to come up with the relevant information, but it seems that its impact is limited.

https://www.rand.org/pubs/research_reports/RRA2977-2.html

osanseviero samusenps clem Dlbk rreed-pha merve


SliceGPT: Compress Large Language Models by Deleting Rows and Columns (2401.15024)

SliceGPT: Compress Large Language Models by Deleting Rows and Columns

https://huggingface.co/papers/2401.15024

Large language models have become the cornerstone of natural language processing, but their use comes with substantial costs in terms of compute and memory resources. Sparsification provides a solution to alleviate these resource constraints, and recent works have shown that trained models can be sparsified post-hoc. Existing sparsification techniques face challenges as they need additional data structures and offer constrained speedup with current hardware. In this paper we present SliceGPT, a new post-training sparsification scheme which replaces each weight matrix with a smaller (dense) matrix, reducing the embedding dimension of the network. Through extensive experimentation, we show that SliceGPT can remove up to 25% of the model parameters (including embeddings) for LLAMA2-70B, OPT 66B and Phi-2 models while maintaining 99%, 99% and 90% zero-shot task performance of the dense model respectively. Our sliced models run on fewer GPUs and run faster without any additional code optimization: on 24GB consumer GPUs we reduce the total compute for inference on LLAMA2-70B to 64% of that of the dense model; on 40GB A100 GPUs we reduce it to 66%. We offer a new insight, computational invariance in transformer networks, which enables SliceGPT and we hope it will inspire and enable future avenues to reduce memory and computation demands for pre-trained models.

osanseviero vladbogo taufiqdp victor CHianogoi omaryshchenko AdinaY merve Locutusque Ji-Ha Saleh


With the Google announcement last week, I think we're now officially the only AI startup out there who has commercial collaborations with all the major cloud providers (AWS, GCP, Azure) and hardware providers (Nvidia, AMD, Intel, Qualcomm,...), making our vision of being the independent and agnostic platform for all AI builders truer than ever!

Let's go!

on1onmangoes osanseviero santiviquez mvaloatto abbasm2 donbale gsarti vladbogo MadElf1337 SivilTaram taufiqdp rreed-pha victor neovalle qnixsynapse andrewrreed vivekrp merve joaogante Bondstreet LucasThil maywell duraad hysts thibaultM


Confidence * may be * all you need.

A simple average of the log probabilities of the output tokens from an LLM might be all it takes to tell if the model is hallucinating.🫨

The idea is that if a model is not confident (low output token probabilities), the model may be inventing random stuff.

In these two papers:
1. https://aclanthology.org/2023.eacl-main.75/
2. https://arxiv.org/abs/2303.08896

The authors claim that this simple method is the best heuristic for detecting hallucinations. The beauty is that it only uses the generated token probabilities, so it can be implemented at inference time ⚡

clem leegao19 osanseviero victor s3nh abbasm2 gsarti rreed-pha stolsvik


TURNA: the biggest Turkish encoder-decoder model up-to-date, based on UL2 architecture, comes in 1.1B params 🐦 😍
The researchers also released models fine-tuned on various downstream tasks including text categorization, NER, summarization and more! 🤯 Great models @onurgu @gokceuludogan @yirmibesogluz @furkanakkurt1618 @uskudarli 👏
Fine-tuned models are in this collection 👉 https://huggingface.co/collections/boun-tabi-LMG/turna-ft-65b3f20aff5235e6cad07c1b
Pre-trained models are in this collection 👉 https://huggingface.co/collections/boun-tabi-LMG/turna-65ad340e5df673eec66e48c7

clefourrier mertyanik clem osanseviero gokceuludogan ridvanpal

osanseviero s3nh ktdilankaya ismaildlml ridvanpal


Sentence Transformers v2.3.0 has been released! It includes several bug fixes, enhanced model loading including custom models & no more unnecessary file downloads, improved performance, a powerful loss function, and much more!

Details:
⬆ Uploading Models to the Hub with `save_to_hub`.
⬇ Downloading Models from the Hub now downloads only necessary files.
⚙ Custom Models (such as https://huggingface.co/jinaai/jina-embeddings-v2-base-de) can now be loaded with `trust_remote_code=True`.
🔍 Models can now be loaded at specific revisions (e.g. commit hashes or git branches).
🖥 Various device fixes; models will now always operate on the device that you specify.
📉 A new "Cached" variant of the powerful Multiple Negatives Ranking Loss allows common hardware to reach performance previously only accessible on multi-gpu clusters.
🐎 Computation time of Community Detection was decreased significantly (7x speedup at 500k sentences :exploding_head:)
🪶 Removed the now unnecessary "torchvision" dependency for a smaller installation.

Check out the full changelog here: https://github.com/UKPLab/sentence-transformers/releases/tag/v2.3.0

I'll be working on much more changes in the near future, so expect more exciting updates. If you encounter any issues, or have any questions or feature requests, don't hesitate to open an issue on the repository: https://github.com/UKPLab/sentence-transformers/issues

osanseviero clem victor chkla santiviquez lbourdois djsull taufiqdp shm141 nickprock hysts


🔥 New LLM leaderboard on the hub: an LLM Hallucination Leaderboard!

Led by @pminervini , it evaluates the propensity of models to *hallucinate*, either on factuality (= say false things) or faithfulness (= ignore user instructions). This is becoming an increasingly important avenue of research, as more and more people are starting to rely on LLMs to find and search for information!
It contains 14 datasets, grouped over 7 concepts, to try to get a better overall view of when LLMs output wrong content.
https://huggingface.co/spaces/hallucinations-leaderboard/leaderboard?logs=container

Their introductory blog post also contains an in depth analysis of which LLMs get what wrong, which is super interesting: https://huggingface.co/blog/leaderboards-on-the-hub-hallucinations

Congrats to the team! 🚀


Current ranking of pre-trained (nonchat) open access LLMs according to the leaderboard. 1-4 are from China-based groups. Does training models with Chinese somehow lead to better metrics? 🤔 WDYT?

1. https://huggingface.co/Qwen/Qwen-72B
2. https://huggingface.co/01-ai/Yi-34B/tree/main
3. https://huggingface.co/internlm/internlm2-20b
4. https://huggingface.co/deepseek-ai/deepseek-llm-67b-base
5. https://huggingface.co/mistralai/Mixtral-8x7B-v0.1

samusenps GregorG abidlabs julien-c admarcosai mishig clefourrier AdinaY shm141 Xhehab

KnutJaegersberg julien-c mhammadkhan aloobun AdinaY zhniu


🔎 DPO hyperparameter search update!

In my previous post (https://huggingface.co/posts/BramVanroy/633544255876795), I indicated how despite high reward accuracies and low losses, my model would sometimes just output repeating random tokens (`/*****/`). There were some useful brainstorms in that thread. I think the dataset is relatively easy for the model, leading it to quickly overfit when the beta is very small, which allows the model to step away further from its initially outputs.

So, I ran a hyperparameter search for learning rate (1e-7 v 5e-7), batch size (32, 64, 96, 128) and most importantly, beta (0.01, 0.1, 0.2, 0.5). You can have a look at the results for yourself here: https://wandb.ai/bramvanroy/dpo-geitje-ultra-hyperparams

Interpreting the result, I'd think that the beta=0.5 is the better choice for this dataset. Reasons:

- markedly higher rewards margins compared to all other betas
- better balance between positive chosen and negative rejected rewards
- log probabilities are not as superbly low as for beta=0.01, which seems too low for this dataset

Of course, that is just purely looking at numbers without running any benchmarks. However, I am hesitant to evaluate all the models on benchmarks and, therefore, literally optimising my hyperparameters on a test set (which is very bad!). So I will just play with some of the most promising models and see which one feels "best" qualitatively.

If you have other insights, thoughts, or opinions, let me know!

osanseviero tomaarsen kashif edbeeching davanstrien philschmid dingo-actual Rijgersberg dvilasuero gabrielmbmb abidlabs julien-c HumanFace Cherran


LLMCheckup: Conversational Examination of Large Language Models via
  Interpretability Tools (2401.12576)

🔍 Today's pick in Interpretability & Analysis of LMs: LLMCheckup: Conversational Examination of Large Language Models via Interpretability Tools by @qiaw99 @tanikina @nfel et al.

Authors introduce LLMCheckup, a conversational interface connecting an LLM to several interpretability tools (feature attribution methods, similarity, counterfactual/rationale generation) allowing users to inquire about LLM predictions using natural language. The interface consolidates several interpretability methods in a unified chat interface, simplifying future investigations into natural language explanations.

📄 Paper: https://huggingface.co/papers/2401.12576
💻 Code: https://github.com/DFKI-NLP/LLMCheckup
🎥 Demo video: https://www.youtube.com/watch?v=ZwN8ZQSXoOU

samusenps osanseviero qiaw99 nfel Citaman compressionsavant Saw


Presenting: SimpleMath

Recently we uploaded on the hub our LATEST and most powerful version of SimpleMath SFT dataset.
Today we are happy to present SimpleMath DPO Pairs, improving further mathematical capabilities on LLM's.

Our first results shows clear improvements on GSM8k, MATHQA, ARC, TQA, MMLU and BBH. Feel free to experiment and generate your own dataset, as we also provide the code to generate them synthetically.

https://huggingface.co/datasets/fblgit/simple-math
https://huggingface.co/datasets/fblgit/simple-math-DPO
https://huggingface.co/fblgit/UNA-34BeagleSimpleMath-32K-v1

samusenps XaevrM sehyun66 fblgit osanseviero victor jtatman


A 1b dense causal language model begins to "saturate" in terms of accuracy around 5 epochs on 1.2T tokens.


Are you working with users and would like to let them edit a text containing highlights? Check out my new Gradio component `HighlightedTextbox`! 🤗

The component can be installed with `pip install gradio_highlightedtextbox` and used seamlessly with other native components in Gradio interfaces. It supports texts with multiple tags and provides some reasonable UX behaviors (tags disappear when an edit is performed inside them). It should be great for UIs to study user editing of annotated LM generations/translations!

Demo: https://huggingface.co/spaces/gsarti/gradio_highlightedtextbox

osanseviero pngwn abidlabs samusenps hutusi victor freddyaboulton

abidlabs osanseviero hutusi julien-c freddyaboulton


👋hi there folks !

check out this Chest X-Ray model from AIMIStanford : https://huggingface.co/spaces/Tonic/CheXRay

thanks to @lunarflu for kicking me a bit to get the examples in there !

would be great to get even more examples and even more downstream functions , so contributions are very welcome, or if you have a dataset source, please do share it in the discussions !

samusenps osanseviero KaiKimera peaceAsh victor clem lunarflu matlok


Exploring the Reasoning Abilities of Multimodal Large Language Models
  (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning (2401.06805)

COCO is "ALL'' You Need for Visual Instruction Fine-tuning (2401.08968)

Thrilled to share some of our recent work in the field of Multimodal Large Language Models (MLLMs).

1⃣ A Survey on Multimodal Reasoning 📚
Are you curious about the reasoning abilities of MLLMs? In our latest survey, we delve into the world of multimodal reasoning. We comprehensively review existing evaluation protocols, categorize the frontiers of MLLMs, explore recent trends in their applications for reasoning-intensive tasks, and discuss current practices and future directions. For an in-depth exploration, check out our paper: https://huggingface.co/papers/2401.06805

2⃣ Advancing Flamingo with InfiMM 🔥
Building upon the foundation of Flamingo, we introduce the InfiMM model series. InfiMM is a reproduction of Flamingo, enhanced with stronger Large Language Models (LLMs) such as LLaMA2-13B, Vicuna-13B, and Zephyr7B. We've meticulously filtered pre-training data and fine-tuned instructions, resulting in superior performance on recent benchmarks like MMMU, InfiMM-Eval, MM-Vet, and more. Explore the power of InfiMM on Huggingface: https://huggingface.co/Infi-MM/infimm-zephyr

3⃣ Exploring Multimodal Instruction Fine-tuning 🖼
Visual Instruction Fine-tuning (IFT) is crucial for aligning MLLMs' output with user intentions. Our research identified challenges with models trained on the LLaVA-mix-665k dataset, particularly in multi-round dialog settings. To address this, we've created a new IFT dataset with high-quality, diverse instruction annotations and images sourced exclusively from the COCO dataset. Our experiments demonstrate that when fine-tuned with this dataset, MLLMs excel in open-ended evaluation benchmarks for both single-round and multi-round dialog settings. Dive into the details in our paper: https://huggingface.co/papers/2401.08968

Stay tuned for more exciting developments.
Special thanks to all our collaborators: @Ye27 @wwyssh @Yongfei @Yi-Qi638 @xudonglin @KhalilMrini @lllliuhhhhggg @Borise @Hongxia

Ye27 wwyssh samusenps lllliuhhhhggg osanseviero victor xudonglin xiaotianhan Tonic yarth69 clem lunarflu

abacaj Tonic clem lunarflu xiaotianhan Yi-Qi638


🔥 New LLM leaderboard on the hub: an LLM Safety Leaderboard!

It evaluates LLM safety, such as bias and toxicity, PII, and robustness, and is powered by DecodingTrust (outstanding paper at Neurips!) 🚀
https://huggingface.co/spaces/AI-Secure/llm-trustworthy-leaderboard

It's great to see such initiatives emerge, trying to understand the risks and biases of LLMs, and I'm hoping other tools will follow. It should be interesting for the community of model builders (whether or not they want uncensored models ^^).

Detailed intro blog: https://huggingface.co/blog/leaderboards-on-the-hub-decodingtrust.

Congrats to the AI Secure team!

clem samusenps monsoon-nlp osanseviero neovalle mc0ps gsarti admarcosai sbrandeis


Gradio 4.16 introduces a new flow: you can hide/show Tabs or make them interactive/non-interactive.

Really nice for multi-step machine learning ademos ⚡

clem samusenps osanseviero gsarti Shivansh000 jiaowobaba02 freddyaboulton


Hello everyone,
This is my first post! I have also decided to release a dataset that I have been keeping private for a while now. I’ve kept it private because I’m not sure if it is actually good or not. I would greatly appreciate it if someone could fine-tune some larger models and evaluate the dataset. Named Hercules-v1.0, it is a turbo-charged version of teknium’s openhermes generated by augmenting its data sources. Learn more in the dataset card: https://huggingface.co/datasets/Locutusque/hercules-v1.0

osanseviero samusenps Locutusque abidlabs clem ujjal13 admarcosai

clem Felladrin afrideva Locutusque maxamed sbrandeis


In 2024, we're expanding from open weights to open EVERYTHING (datasets, training scripts,...).

Excited to see this dataset release in French by @Pclanglais  @carbonbasedLLM @anastasiastasenko:
https://huggingface.co/datasets/PleIAs/French-PD-Newspapers

"To give you an idea of the size, the full French Wikipedia is about 2 billon words. This is 40 times larger."

Maani Pclanglais afrideva tomaarsen osanseviero samusenps glenn-parham mlabonne ucalyptus abidlabs Sylvestre JaiSurya anastasiastasenko mvaloatto sparkycollier linoyts MexIvanov AdinaY brishtiteveja


Diffuse to Choose: Enriching Image Conditioned Inpainting in Latent
  Diffusion Models for Virtual Try-All (2401.13795)

Diffuse to Choose

Enriching Image Conditioned Inpainting in Latent Diffusion Models for Virtual Try-All

paper page: https://huggingface.co/papers/2401.13795

Diffuse to Choose (DTC) allows users to virtually place any e-commerce item in any setting, ensuring detailed, semantically coherent blending with realistic lighting and shadows.

blanchon clem mvaloatto Danyray101 osanseviero samusenps ucalyptus Shivansh000


The Calibration Gap between Model and Human Confidence in Large Language
  Models (2401.13835)

🔍 Today's pick in Interpretability & Analysis of LMs: The Calibration Gap between Model and Human Confidence in Large Language Models by @coolprof H. Tejeda A. Kumar @Cbelem @skarny et al.

This work involves an experimental study to assess human confidence in LLM responses to multiple-choice MMLU questions based on explanations the LLM provides together with the selected answer. The authors experiment with altering the model prompt to reflect the actual prediction confidence in models’ explanations, showing improved calibration for users’ assessment of LLM’s reliability and a better ability to discriminate between correct and incorrect answers. These results suggest the importance of further research on the impact of automatic explanations on users’ perception.

📄 Paper: https://huggingface.co/papers/2401.13835

osanseviero clem victor clefourrier afrideva skarny samusenps aakritikumar adhisetiawan sbrandeis


If you have to choose one small base language model <=3B for ChatML Code Assistant (SFT+DPO) to validate the approach on the dataset and tune hyperparams, so later retrain with a larger base model like Mistral/Mixtral, what model would you pick?
🧵

afrideva Noahosmonth joshpxyne sbrandeis


Hi everyone,
For my first post, I'm announcing a big release (in multiple ways): probably the largest open corpus in French to date, with 85 billion words in the public domain.
The dataset has been prepared in collaboration with Benoît de Courson and Benjamin Azoulay from Gallicagram (https://shiny.ens-paris-saclay.fr/app/gallicagram). Gallicagram is a major cultural analytics project in French, the open and better version of ngram viewer for large scale search of word and ngram occurrences.
The corpus is made of two different dataset for monographs (16B words) https://huggingface.co/datasets/PleIAs/French-PD-Newspapers and newspapers/periodicals (69B) https://huggingface.co/datasets/PleIAs/French-PD-Newspapers Along with the full text it also includes core provenance metadata.
Beyond research in digital humanities, the corpus can also be used to training open and reproducible LLMs. Being in the public domain means it can be released everywhere in any shape without restrictions.
The corpus is not perfect: digitization of cultural heritage is challenging and, especially for newspapers, we tackle with layout issues and a significant rate of optical character recognition mistake. Our conviction is that releasing corpus as a commons is the best way to improve on this. Sharing is caring.

anastasiastasenko davanstrien julien-c osanseviero clem victor Tonic Pclanglais blanchon bgowan philschmid kramp Wauplin clefourrier afrideva samusenps KaiKimera kheopss yjernite sbrandeis

davanstrien julien-c osanseviero clem kramp vicgalle Tonic blanchon philschmid damerajee danielus

neovalle julien-c osanseviero clem Tonic samusenps damerajee


My TED talk is finally live!! I proposed the recipe for the "Foundation Agent": a single model that learns how to act in different worlds. LLM scales across lots and lots of texts. Foundation Agent scales across lots of lots of realities. If it is able to master 10,000 diverse simulated realities, it may well generalize to our physical world, which is simply the 10,001st reality.

Why do we want a single Foundation Agent instead of many smaller models? I'd like to quote the idea from my friend Prof. Yuke Zhu's CoRL keynote talk. If we trace each AI field's evolution, we'd find this pattern:

Specialist -> Generalist -> Specializing Generalist

And the "specialized generalist" is often way more powerful than the original specialist. Just like distilled versions of LlaMA are way better than custom-built NLP systems 5 years ago.

TED talks do not have teleprompters!! All I have is a "confidence monitor" at my foot, showing only the current slide and timer. That means I need to memorize the whole speech. It sounds intimidating at first, but turns out to be the best way to connect with the audience and deliver the ideas right to heart.

Happy to share my slides with all of you! https://drive.google.com/file/d/1NSY6MxMu3OPQ4U6hx0OxPq7EQB5XTcAG/view?usp=sharing

The video is only 10 min. I promise it's well worth your time!
https://www.ted.com/talks/jim_fan_the_next_grand_challenge_for_ai

truong-xuan-linh s3nh gsarti samusenps osanseviero clem victor julien-c akhaliq juancopi81 afrideva frankdata

samusenps osanseviero clem julien-c akhaliq sbrandeis


Do you have a hidden massive storage leak thanks to HF hub models and datasets revisions adding up and not getting automatically deleted?

Here is how to delete all old revisions and only keeping `main` in a few quick steps and no tedious manual editing.

In terminal A:
```
$ pip install huggingface_hub["cli"] -U
$ huggingface-cli delete-cache --disable-tui
File to edit: /tmp/tmpundr7lky.txt
0 revisions selected counting for 0.0. Continue ? (y/N)
```
Do not answer the prompt and proceed with my instructions.

(note your tmp file will have a different path, so adjust it below)

In terminal B:
```
$ cp /tmp/tmpedbz00ox.txt cache.txt
$ perl -pi -e 's|^#(.*detached.*)|$1|' cache.txt
$ cat cache.txt >>  /tmp/tmpundr7lky.txt
```
The perl one-liner uncommented out all lines that had `(detached)` in it - so can be wiped out. And then we pasted it back into the tmp file `huggingface-cli` expects to be edited.

Now go back to terminal A and hit: N, Y, Y, so it looks like:

```
0 revisions selected counting for 0.0. Continue ? (y/N) n
89 revisions selected counting for 211.7G. Continue ? (y/N) y
89 revisions selected counting for 211.7G. Confirm deletion ? (Y/n) y
```
Done.

If you messed up with the prompt answering you still have `cache.txt` file which you can feed again to the new tmp file it'll create when you run `huggingface-cli delete-cache --disable-tui` again.

For more details and additional techniques please see https://github.com/stas00/ml-engineering/tree/master/storage#huggingface-hub-caches

vicgalle cahlen osanseviero not-lain samusenps clem victor Wauplin afrideva sethuiyer sbrandeis


New in Gradio 4.16.0 - Galleries as Input 🖼

Now your users can upload multiple images as input to your AI application and view them in a slick gallery!

Attached is a demo of how this new feature can be used in a photomaker-type application: https://huggingface.co/spaces/TencentARC/PhotoMaker

Shout out @abidlabs and @akhaliq who proposed this feature after seeing some of the workarounds gradio developers were using in the wild to upload multiple images.

The gradio team works hard to stay up to date with the latest trends in AI! If there's something missing from the library, file an issue on github! https://github.com/gradio-app/gradio/issues

kramp ysharma sbrandeis samusenps afrideva baxin mrfakename Sattineez toyxyz


✨ Excited to release `gradio` 4.16. New features include:

🐻❄ Native support for Polars Dataframe
🖼 Gallery component can be used as an input
⚡ Much faster streaming for low-latency chatbots
📄 Auto generated docs for custom components

... and much more! This is HUGE release, so check out everything else in our changelog:  https://github.com/gradio-app/gradio/blob/main/CHANGELOG.md

clem sbarman25 osanseviero gsarti samusenps ysharma freddyaboulton


Google + Hugging Face + Open-Source AI = 🔥🔥🔥

https://huggingface.co/blog/gcp-partnership

https://finance.yahoo.com/video/google-hugging-face-alliance-spur-173016882.html

https://www.theverge.com/2024/1/25/24050445/google-cloud-hugging-face-ai-developer-access

https://www.bloomberg.com/news/articles/2024-01-25/google-to-team-up-with-startup-hugging-face-to-host-ai-software

https://www.reuters.com/technology/google-cloud-partners-with-hugging-face-attract-ai-developers-2024-01-25/

EmilyWitko osanseviero vladbogo neovalle FranzNo mrfakename radames akhaliq not-lain afrideva Sylvestre

osanseviero Paul-HF vicgalle akhaliq koisose stas not-lain taufiqdp sbrandeis

mrfakename akhaliq not-lain kramp mvaloatto


We just released the first Text2SQL model for DuckDB 🦆🧠
You can try it out directly here :
https://huggingface.co/spaces/motherduckdb/DuckDB-NSQL-7B

osanseviero samusenps lunarflu clem julien-c not-lain lhoestq Johnnycadelover lysandre bgowan afrideva meetrais jarvisx17 rreed-pha cnmoro severo notsahil sbrandeis

osanseviero lunarflu clem julien-c lhoestq koisose lysandre kramp mrfakename

osanseviero lunarflu julien-c lysandre Tonic afrideva rreed-pha


hey there folks , work in progress, but basically celebrating the release of `whisperspeech`

just :

```bash
pip install whisperspeech
```
to get started and check out my demo to do multilingual text to speech including making voice prints using `whisperspeech`reverse engineering of whisper here : https://huggingface.co/spaces/Tonic/whisperspeech
and the model card here : https://huggingface.co/collabora/whisperspeech

i met collabora on LAION check out LAION here :
https://huggingface.co/laion

lunarflu samusenps clem Johnnycadelover cahlen afrideva emilios rreed

julien-c osanseviero lunarflu clem rreed


I've just published the 23rd edition of the satellite-image-deep-learning newsletter to 8,188 subscribers

This edition: METEOR, Seeing the roads through the trees 🌴, A New Learning Paradigm for Foundation Model-based Remote Sensing Change Detection & Globe230k dataset

https://www.satellite-image-deep-learning.com/p/new-discoveries-23

andysalerno Tonic osanseviero samusenps clem abidlabs sbrandeis

Ch-Usman marinaretik Tonic osanseviero abidlabs


Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic
  Image Restoration In the Wild (2401.13627)

Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild

paper page: https://huggingface.co/papers/2401.13627

introduce SUPIR (Scaling-UP Image Restoration), a groundbreaking image restoration method that harnesses generative prior and the power of model scaling up. Leveraging multi-modal techniques and advanced generative prior, SUPIR marks a significant advance in intelligent and realistic image restoration. As a pivotal catalyst within SUPIR, model scaling dramatically enhances its capabilities and demonstrates new potential for image restoration. We collect a dataset comprising 20 million high-resolution, high-quality images for model training, each enriched with descriptive text annotations. SUPIR provides the capability to restore images guided by textual prompts, broadening its application scope and potential. Moreover, we introduce negative-quality prompts to further improve perceptual quality. We also develop a restoration-guided sampling method to suppress the fidelity issue encountered in generative-based restoration. Experiments demonstrate SUPIR's exceptional restoration effects and its novel capacity to manipulate restoration through textual prompts.


Model Editing Can Hurt General Abilities of Large Language Models (2401.04700)

🔍 Today's pick in Interpretability & Analysis of LMs: Model Editing Can Hurt General Abilities of Large Language Models by J.C. Gu et al.

This work raises concerns that gains in factual knowledge after model editing can result in a significant degradation of the general abilities of LLMs. The authors evaluate 4 popular editing methods on 2 LLMs across eight representative tasks, showing model editing does substantially hurt model general abilities. A suggestion is made to prioritize improvements in LLMs' robustness, developing more precise editing methods, and better evaluation benchmarks.

📄 Paper: https://huggingface.co/papers/2401.04700
💻 Code: https://github.com/JasonForJoy/Model-Editing-Hurt

merve vicgalle osanseviero Dlbk clem samusenps lunarflu


Can you merge models of different sizes? ⚗

Well, yes, if the models are somewhat compatible. Here is an experiment I did. I wanted to merge two of the best performing models: https://huggingface.co/mlabonne/NeuralBeagle14-7B and https://huggingface.co/jeonsworld/CarbonVillain-en-10.7B-v4

Here is my recipe:
1. Expand the layers of NeuralBeagle to 10.7B ala frankenmerge.
2. DPO-tune the previous model with a high-quality preference dataset, https://huggingface.co/datasets/argilla/distilabel-intel-orca-dpo-pairs
3. Merge the previous model with CarbonVillain (needs —allow-crimes in mergekit! 🔪)

And here is the resulting model, CarbonBeagle-11B, which ranked top in the leaderboard for its size class:
https://huggingface.co/vicgalle/CarbonBeagle-11B

vicgalle victor samusenps NeuralNovel itsn78 merve Dlbk clem antiven0m lunarflu Svngoku Makya mlabonne tryumanshow

osanseviero itsn78 merve clem cbensimon lunarflu luqmanxyz vicgalle Makya roinaveiro


Lumiere: A Space-Time Diffusion Model for Video Generation (2401.12945)

LUMIERE

A Space-Time Diffusion Model for Video Generation

paper page: https://huggingface.co/papers/2401.12945

Demonstrate state-of-the-art text-to-video generation results, and show that our design easily facilitates a wide range of content creation tasks and video editing applications, including image-to-video, video inpainting, and stylized generation.

osanseviero victor s3nh taufiqdp samusenps podevi merve clem blanchon

osanseviero wssb mvaloatto merve clem tengomucho


GPU Poor POV: Willingness of Customization

I love to use libraries in which you can customize a lot of things. Chromadb is my choice of db if it comes to store embeddings. Te cool feature is that you can define your own embeddings function which can be called on every chromadb collection initialisation or creation. It is useful because sometimes we want to use  different prompts, different models, and it can be easily written as inheritence from EmbeddingFunction class.

Edit:

My CustomEmbeddingFunction can be found here:
https://gist.github.com/s3nh/cfbbf43f5e9e3cfe8c3e4e2f0d550b80

and you can use it by initializing or calling the chroma collection.

```python

import chromadb
from your_custom_fn import CustomEmbeddingFunction
class ChromaStorage:
    def __init__(self, config):
        self.config = config
        self.client = self.init_client()
        self.embedding_function = CustomEmbeddingFunction()

    def check_config(self):
        assert os.path.exists(self.config.path), ValueError('Provided path does not exists!!')

    def init_client(self):
        return chromadb.PersistentClient(path = self.config.path,)

    def init_collection(self, name: str):
        return self.client.get_or_create_collection(name = name, embedding_function = self.embedding_function)
```

victor julien-c samusenps NeuralNovel krinal clem s3nh lunarflu KnutJaegersberg tawfikgh Ji-Ha

osanseviero victor julien-c NeuralNovel clem


👋 Hi there folks,

I launched my first competition !

Goal : Use AI to beat the Math Olympics within the set time

Basically we're looking for adventurous teams and individuals to make a common submission to the AI Math Olympics by the MLCommons.

Althought the ultimately there can only be one winner and there must always be a winner, the ultimate goal is to get together for a common solution.

check it out here :
https://huggingface.co/spaces/Tonic1/mathathon

s3nh samusenps osanseviero boinc victor Solshine clem lunarflu Ji-Ha


🏅 New top model on the GAIA benchmark!

Called FRIDAY, it's a mysterious new autonomous agent, which got quite good performances on both the public validation set *and* the private test set.
It notably passed 10 points for the val and 5 points for the test set on our hardest questions (level 3): they require to take arbitrarily long sequences of actions, use any number of tools, and access the world in genera! ✨

The GAIA benchmark evaluates next-generation LLMs (LLMs with augmented capabilities due to added tooling, efficient prompting, access to search, etc) and was co authored by @gregmialz @ThomasNLG @ylecun @thomwolf and myself: https://huggingface.co/spaces/gaia-benchmark/leaderboard

gregmialz osanseviero Tonic m-ric clem julien-c kahris

osanseviero Tonic sharad mariagrandury clem diwank m-ric


From Understanding to Utilization: A Survey on Explainability for Large
  Language Models (2401.12874)

🔍 Today's pick in Interpretability & Analysis of LMs: From Understanding to Utilization: A Survey on Explainability for Large Language Models by H. Luo and L. Specia

This survey summarizes recent works in interpretability research, focusing mainly on pre-trained Transformer-based LMs. The authors categorize current approaches as either local or global and discuss popular applications of LM interpretability, such as model editing, enhancing model performance, and controlling LM generation.

📄 Paper: https://huggingface.co/papers/2401.12874

osanseviero cihan30 samusenps aigclab clem lunarflu


Making gradio's auto-reload leaner and more robust 💪

Auto-reload is one of the coolest (and maybe underrated) features of gradio in my opinion. It automatically detects changes in your python app file and hot swaps your backend and frontend without restarting the server. I may be biased, but that's a way nicer experience than most other reload mode features out there.

This is all possible due to novel use of python's built in importlib module. Since this is a non-standard use of the module, our users have encountered some rough edges in the wild.

I'm happy to announce two improvements that will be out in the next release of gradio. They were both made possible by collaboration in the open source community.

1. First, gradio's reload mode now works with the `python-dotenv` library. Getting to the bottom of this one took a lot of sleuthing by our users and the fix got merged and released into the `python-dotenv` package last night! Thanks to @theskumar, maintainer of python-dotenv, for all the help.
2. Second, gradio's reload mode now consumes an order of magnitude less CPU. Depending on how many files are in your source directory, you may see between 2x to 10x less CPU utilization. Thanks to @velaia for filing the issue and @abidlabs for the review!

More improvements to reload mode are planned. Stay tuned!

clem abidlabs osanseviero victor vladbogo gsarti samusenps radames chansung akashicmarga mathiasn1 Dlbk Wauplin kramp clefourrier velaia hysts

abidlabs osanseviero clem radames chansung ashwinlimaye


🕵 Looking for DPO experts!

I have a dataset that is gpt-4-turbo as chosen and a lower performing model as rejected. The objective should therefore be fairly easy because the two are easy to discern. As a consequence, the model achieves very low losses (0.021 train; 0.013 validation) and high reward accuracies (0.995). **However**, when using the model in practice, it often detoriates after the first one or two tokens and continuously outputs sequences of `/*****/`. So despite the good performance on the DPO objective and strong scores on the validation set (no overfitting), something seems to go wrong. Perhaps the outputs are too different and the task is too easy, in which case DPO is not useful. But why then would the model start hallucinating and repeating the same token over and over again?

Any thoughts? Any suggestions to get around this? All discussions are welcome!

mrfakename davanstrien clem osanseviero lewtun Dlbk lilsyoss gsarti julien-c clefourrier lunarflu larryvrh


✅New paper to ensure valid LLM output with SOTA LLMs like GPT4 by mixing it with OSS LLMs

Paper: arxiv.org/abs/2401.09967

Great paper showing how strong proprietary AI like #GPT4 can be paired with #OSS LLM to ensure LLM output validity, e.g. valid JSON.

Many devs complain that #LLMs cannot be reliably used in production if the output is not valid, for instance, if one wants to use LLMs to generate SQL queries or JSON, it is crucial that the output is valid.

Frameworks have arisen to constrain the outputs of the LLM to follow some constraints, like outlines (https://github.com/outlines-dev/outlines), but they assume access to logits.

This makes them incompatible with proprietary LLMs like GPT4 that don’t share logits, so one can only use open-source LLMs that are much less performant.

This paper shows how can use powerful proprietary LLMs like GPT4 to create a first unconstrained sketch and refine it using an OSS model like Llama 2 where logits are accessible, to rewrite the sketch following some specific constraints.

They show that GPT4 Precision can be increased by 14% (43% before, 57% after), by boosting it with constrained output on information extraction on Wiki-NRE!


What's the best way to fine-tune open LLMs in 2024? Look no further! 👀 I am excited to share “How to Fine-Tune LLMs in 2024 with Hugging Face” using the latest research techniques, including Flash Attention, Q-LoRA, OpenAI dataset formats (messages), ChatML, Packing, all built with Hugging Face TRL. 🚀

It is created for consumer-size GPUs (24GB) covering the full end-to-end lifecycle with:
💡Define and understand use cases for fine-tuning
🧑🏻💻 Setup of the development environment
🧮 Create and prepare dataset (OpenAI format)
🏋♀ Fine-tune LLM using TRL and the SFTTrainer
🥇 Test and evaluate the LLM
🚀 Deploy for production with TGI

👉  https://www.philschmid.de/fine-tune-llms-in-2024-with-trl

Coming soon: Advanced Guides for multi-GPU/multi-Node full fine-tuning and alignment using DPO & KTO. 🔜

andrewrreed Jofthomas victor stefan-jo s3nh osanseviero luqmanxyz philschmid kramp samusenps Csplk lunarflu julien-c hamedbabaeigiglou harpreetsahota dhuynh95 clem CultriX afrideva brekk andysalerno PriyankSisodia azimuth3d nuatmochoi mathiasn1 ashwinlimaye AdinaY vicgalle pardeep merve adamo1139 madroid mikr omaryshchenko alielfilali01 patchie2 celsowm Luciano santyzenith sbrandeis ramkrish120595 Utkarsh736

BramVanroy clem osanseviero Norod78 hayoungpark azimuth3d mvonwyl pardeep KrishnaKaasyap merve lunarflu madroid zappa2005 Kamaljp mikr vivek37kumar raychen26 gzy02 InvictusRudra ddahlmeier


CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding
  Benchmark (2401.11944)

CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark

paper page: https://huggingface.co/papers/2401.11944

As the capabilities of large multimodal models (LMMs) continue to advance, evaluating the performance of LMMs emerges as an increasing need. Additionally, there is an even larger gap in evaluating the advanced knowledge and reasoning abilities of LMMs in non-English contexts such as Chinese. We introduce CMMMU, a new Chinese Massive Multi-discipline Multimodal Understanding benchmark designed to evaluate LMMs on tasks demanding college-level subject knowledge and deliberate reasoning in a Chinese context. CMMMU is inspired by and strictly follows the annotation and analysis pattern of MMMU. CMMMU includes 12k manually collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering, like its companion, MMMU. These questions span 30 subjects and comprise 39 highly heterogeneous image types, such as charts, diagrams, maps, tables, music sheets, and chemical structures. CMMMU focuses on complex perception and reasoning with domain-specific knowledge in the Chinese context. We evaluate 11 open-source LLMs and one proprietary GPT-4V(ision). Even GPT-4V only achieves accuracies of 42%, indicating a large space for improvement. CMMMU will boost the community to build the next-generation LMMs towards expert artificial intelligence and promote the democratization of LMMs by providing diverse language contexts.

osanseviero clefourrier lunarflu clem afrideva walzay AdinaY


Universal Neurons in GPT2 Language Models (2401.12181)

🔍 Today's pick in Interpretability & Analysis of LMs: Universal Neurons in GPT2 Language Models by @wesg @thorsley @luminolblue T.R. Kheirkhah Q. Sun, @willhath @NeelNanda D. Bertsimas

This work investigates the universality of individual neurons across GPT2 models trained from different initial random seeds, starting from the assumption that such neurons are likely to exhibit interpretable patterns. The authors find 1-5% of neurons showing high correlation across five model seeds, i.e. consistently activating for the same inputs. In particular, those neurons can be grouped into families exhibiting similar functional roles, e.g. modulating the next token prediction entropy, controlling the output norm of an attention head, and promoting/suppressing vocabulary elements in the prediction. Finally, universal neurons are often observed to form antipodal pairs, conjectured to improve the robustness and calibration of model predictions via ensembling.

📄 Paper: https://huggingface.co/papers/2401.12181

merve osanseviero wesg lunarflu abidlabs afrideva LiraMirui Aryanne sbrandeis

s3nh merve osanseviero wesg clem abidlabs lunarflu


Hi Community!
I'm ecstatic to announce Flutter SDK for HuggingFace Inference APIs. The project is open-source and can be found at https://github.com/shivance/huggingface.dart . I have also created a sample Flutter App that does language translation using HuggingFace's Helsinki-NLP/opus-mt-ru-en" model. Below is the video demo.

## Why Flutter?

Flutter is an open-source UI software development kit created by Google. It is used to develop cross-platform applications from a single codebase for any Web Browser, Android, iOS, Linux, macOS, and Windows.

Over the past few years, the framework has grown a lot of popularity.

## Mission
This is a small step-stone to bring Open Source Machine Learning to the edge and thus make AI more accessible. This project, at the time of writing, supports only NLP APIs, I'll slowly add other modalities too.

samusenps afrideva osanseviero tomaarsen Wauplin merve lunarflu clem hereral kevineen jaecabrera valfcn

samusenps osanseviero merve chansung lunarflu clem


𝗛𝗼𝘄 𝘄𝗲 𝗺𝗮𝗱𝗲 𝗚𝗿𝗮𝗱𝗶𝗼 𝗳𝗮𝘀𝘁𝗲𝗿 𝗯𝘆... 𝘀𝗹𝗼𝘄𝗶𝗻𝗴 𝗶𝘁 𝗱𝗼𝘄𝗻!

About a month ago, @oobabooga (who built the popular text generation webui) reported an interesting issue to the Gradio team. After upgrading to Gradio 4, @oobabooga noticed that chatbots that streamed very quickly had a lag before their text would show up in the Gradio app.

After some investigation, we determined that the Gradio frontend would receive the updates from the backend immediately, but the browser would lag before rendering the changes on the screen. The main difference between Gradio 3 and Gradio 4 was that we migrated the communication protocol between the backend and frontend from Websockets (WS) to Server-Side Events (SSE), but we couldn't figure out why this would affect the browser's ability to render the streaming updates it was receiving.

After diving deep into browsers events, @aliabid94 and @pngwn made a realization: most browsers treat WS events (specifically the `WebSocket.onmessage` function) with a lower priority than SSE events (`EventSource.onmessage` function), which allowed the browser to repaint the window between WS messages. With SSE, the streaming updates would stack up in the browser's event stack and be prioritized over any browser repaint. The browser would eventually clear the stack but it would take some time to go through each update, which produced a lag.

We debated different options, but the solution that we implemented was to introduce throttling: we slowed down how frequently we would push updates to the browser event stack to a maximum rate of 20/sec. Although this seemingly “slowed down” Gradio streaming, it actually would allow browsers to process updates in real-time and provide a much better experience to end users of Gradio apps.

See the PR here: https://github.com/gradio-app/gradio/pull/7084

Kudos to @aliabid94 and @pngwn for the fix, and to @oobabooga and @pseudotensor for helping us test it out!

kramp osanseviero pngwn Ifeanyi xpgx1 BramVanroy takeraparterer samusenps clem phanes gate369 fffiloni sbarman25 taufiqdp akhaliq ysharma davanstrien gsarti lunarflu julien-c merve hysts freddyaboulton

clem akhaliq ysharma kramp santiviquez gsarti lunarflu julien-c

takeraparterer clem akhaliq ysharma gsarti lunarflu julien-c


Re-posting @karpathy's blogpost here because it's down on https://karpathy.github.io/2024/01/21/selfdriving-agi. What do you all think?

elpreneurAbdo RodolpheD Paul-HF SaylorTwift stefan-jo arnaudliotta lunarflu julien-c merve radames pierrci

osanseviero chansung samu lunarflu julien-c merve


Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data (2401.10891)

Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data

paper page: https://huggingface.co/papers/2401.10891
demo: https://huggingface.co/spaces/LiheYoung/Depth-Anything

Depth Anything is trained on 1.5M labeled images and 62M+ unlabeled images jointly, providing the most capable Monocular Depth Estimation (MDE) foundation models with the following features:

zero-shot relative depth estimation, better than MiDaS v3.1 (BEiTL-512)

zero-shot metric depth estimation, better than ZoeDepth

optimal in-domain fine-tuning and evaluation on NYUv2 and KITTI

Norod78 jgitsolutions clem Chunte alazro victor samusenps XaevrM NeuralNovel Kutches


GPU Poor POV: Dont be Afraid :D

Sometimes we dont want to do something because of low self esteem,
I ofter hear 'its to hard for me','i am not an expert','i do not know how to do it', etc. These  words are  never the truth,  we should not be afraid and try to build something because there is no additive value without a failure.

Same things comes in LLMs, there is a lot of fancy words happening, but whats is more important is that there are also  people who are constantly building so other can build.  Diving into finetuning LLMs is incredibly simple if we assume using axolotl library and pretrains stored on huggingface.

All we need is an idea, our GPU Poor desktop or colab notebooks and these steps:
```
git clone https://github.com/OpenAccess-AI-Collective/axolotl
cd axolotl

pip3 install packaging
pip3 install -e '.[flash-attn,deepspeed]'
```
After installation process we can go to examples, and modify configs to our own needs.
Lets jump into
```
axolotl\examples\llama-2\qlora.yml
```
and change
```
base_model: NousResearch/Llama-2-7b-hf
```
to
```
base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
```
choose dataset from huge amounts of dataset that are possible to use from hf.co/datasets and tweak additional params like batch_size, number of epochs, how often do we want to save our model and many more (which I wont focus on rn).
Then,
```
accelerate launch -m axolotl.cli.train examples/llama-2/qlora.yml
```
Will allow to start the finetuning process on structure defined strictly by you.  After finetuning, model will be saved in path provided in config, and you can check out if it performs better than  the base one. Or even you can put it on llm Leaderboard to check if we do not have new SOTA :)
Have fun and have a great day <3

KnutJaegersberg Csplk miesnerjacob victor alazro becstar samusenps ameerazam08 NeuralNovel pro7357 s3nh Tanvir1337 Solshine merve aardoi joujiboi danielus Ji-Ha


Towards Best Practices of Activation Patching in Language Models:
  Metrics and Methods (2309.16042)

🔍 Today's pick in Interpretability & Analysis of LMs: Towards Best Practices of Activation Patching in Language Models: Metrics and Methods by @m0pp11 and @NeelNanda

This work systematically examines the impact of methodological details in activation patching, a popular technique with causal guarantees to quantify the importance of model components in driving model predictions. Authors' recommendations include 1) using in-distribution counterfactual prompts instead of noise/zeroing to mitigate the OOD problem, 2) using logins instead of probabilities as evaluation metrics to enable the discovery of model components with negative influence on predictions, 3) accounting for interaction factors across layers when performing multi-layer patching; and 4) experiment with corrupting different prompt tokens to verify their agreement in the resulting discovered circuits.

📄 Paper: https://huggingface.co/papers/2309.16042

clem victor osanseviero lunarflu clefourrier m0pp11


Update on the Newsletter of 🤗 Daily Paper

Automatic Korean translation is integrated. In the newspaper, "KO" links appear, and it will bring you to the translated version of full paper. This is done with the following workflow.

1. Grasp the list of arXiv IDs from 🤗 Daily Paper API
2. Distribute a number of sub-list of arXiv IDs to VMs (possibly spot instances since the job ends shortly)
3. Commit & push the translated paper in HTML to the designated GitHub repository
4. Newsletter will include the links to the HTML of each paper

Job distribution to a number of VMs are super easily done with [dstack]( https://dstack.ai/ ), and the translation sub-workflow is done through 1) download PDF of each paper with arxiv-dl package, 2) PDF => text with nougat-ocr package, 3) a custom trained model( https://huggingface.co/nlp-with-deeplearning/enko-t5-small-v0 ) in 🤗 transformers to translate the English text into Korean line by line, and 4) reformat the translation into HTML.

Many people in Korea are not fluent in English but want to learn about new stuff in AI, so they usually use Google Translate or other services. This is why I made this feature for easier and direct access to the SOTA knowledge.

Are there other countries with the similar needs? If so, it would be wonderful to cooperate to support more languages. Please reach out anyone is interested in this.

PS; I always wanted to show the usefulness of open ML models by building a well working end to end product, and this newsletter shows it by featuring T5ForConditionalGeneration (translation), SOLAR LLM (summarization).

if you want to sub to the newsletter
: https://groups.google.com/g/hf-daily-paper-newsletter

if you want to look into the source codes
: https://github.com/deep-diver/hf-daily-paper-newsletter

s3nh peterschmidt85 aidystark afrideva clem victor samusenps maywell lunarflu julien-c shrijayan ysharma dillfrescott JUNGU


Migrated all my GPU consuming Spaces to ZERO, it was super easy to do so (add three lines of code and voila!) and the start-up time decreased dramatically as well 💜

s3nh NeuralNovel SharathChenna samusenps jkopz akhaliq Chunte chansung aidystark gbharti osanseviero abidlabs clem KnutJaegersberg koisose Tanvir1337 awinml julien-c pierrci clefourrier Norod78 Sentdex mikr cbensimon tsukumijima mahiatlinux asad Ryukijano taesiri Nymbo leeloolee


GPU Poor POV: My storytelling choices of the week

Its end of the week, I decided to summarize my observations in community based LLMs and mention few models in specific area which are very interesting and has capability to create some insightful stories despite of its relatively lightweight form.

I personally did not use LLMs in my daily routine to tasks like function calling, parsing or assist in code writing. What I tried to use for is storytelling, because it always amaze me how different these models comes to different preferred tasks.

How this model are able to generalize the stories and sometimes, how high level of creativity they carry.

https://huggingface.co/BlueNipples/DaringLotus-v2-10.7b its main target is to generate prose. Quoting the author 'It shares it's good prose, and relatively decent coherency, being a little bit more on the side of prose, and a little bit less on the side of coherency. I like this model for generating great prose if I feel like regening a bit. '

https://huggingface.co/NeuralNovel/Aeryth-7B-v0.1
great work by @NeuralNovel , I really like how flexible this model is, there is no strict focus on a certain role, so definitely worth a try. Would love to hear more about dataset on which was trained, afaik is private rn. best suited for Science Fiction, History & Romance genres due to the training data used.

And the last one for today is https://huggingface.co/FPHam/Sydney_Pirate_Mistral_7b @FPHam work always amaze me how the models are able to stick to provided role. awesome work as always, Ill for sure use this model to generate some interesting stories.

I know that hype train is going fast but as I observe people here on huggingface are creating really creative models which are for sure worth to try. Have a great day <3

KnutJaegersberg osanseviero FlareRebellion NeuralNovel Dlbk abdd199719 merve samusenps NovoCode HuggyMonkey dvilasuero abidlabs clem s3nh pro7357 Tanvir1337

NeuralNovel samusenps NovoCode afrideva clem Tanvir1337


I've prepared  a Google Colab notebook  which allows you to play with interpolating between different people using IP-Adapter SDXL Face-ID Plus.

```

#Prepare a list t of num_of_results values between 0 and 1
t_space = torch.linspace(0, 1, num_of_results)
for t in tqdm(t_space):
    mix_factor = t.item()
    # interpolate between the two face images
    image = (image1 * (1 - mix_factor) + image2 * mix_factor).astype(np.uint8)
    # interpolate between the two face embedding
    faceid_embeds = torch.lerp(faceid_embeds1, faceid_embeds2, t)
   #generate interpolated result
    images = ip_model.generate(prompt=prompt, negative_prompt=negative_prompt, face_image=image, faceid_embeds=faceid_embeds, shortcut=v2, num_samples=2, scale=scale, s_scale=s_scale, guidance_scale=guidance_scale, width=width, height=height, num_inference_steps=steps, seed=seed)

```

Link to notebook:
https://colab.research.google.com/#fileId=https%3A//huggingface.co/datasets/Norod78/face_id_v2_test_code/blob/main/norod78_faceid_sdxl_plus_v2_test.ipynb

Link to Face-ID Repo:
https://huggingface.co/h94/IP-Adapter-FaceID

Link to all sorts of generated examples (Use the file tab):
https://huggingface.co/datasets/Norod78/face_id_v2_test_code/tree/main/sdxl_plus_v2_outputs

osanseviero linoyts clem merve HuggyMonkey tonyassi

s3nh clem kevinstonge samusenps tonyassi fahim1233


There's a lot of interest in machine learning models that generate 3D objects, so Gradio now supports previewing STL files natively in the  `Model3D` component. Huge thanks to `Monius` for the contribution 🔥🔥

samusenps Maykeye osanseviero AIIAR Dlbk clem rreed linoyts johko Chunte HuggyMonkey lunarflu ciCic AgastyaPatel marianbasti goendalf666 sugatoray conceptron

samusenps AIIAR clem abidlabs taufiqdp NeuralNovel Chunte lunarflu


🌳 Model Family Tree

Merging models has become a powerful way to compress information and build powerful models for cheap. Right now, the process is still quite experimental: which models to merge? which parameters should I use? We have some intuition but no principled approach.

I made a little tool to make things a little clearer. It allows you to visualize the family tree of any model on the Hub. It also displays the type of license they use: permissive (green), noncommercial (red), and unknown (gray). It should help people select the right license based on the parent models.

In addition, I hope it can be refined to extract more information about these models: do models from very different branches work better when merged? Can we select them based on the weight difference? There are a lot of questions to explore in this new space. :)

Here's a link to the colab notebook I made: https://colab.research.google.com/drive/1s2eQlolcI1VGgDhqWIANfkfKvcKrMyNr
If you want to know more about model merging or build you own merges, here's the article I wrote about this topic: https://huggingface.co/blog/mlabonne/merge-models

afrideva MaziyarPanahi Jofthomas samusenps flemmingmiguel osanseviero vince62s ucalyptus victor mrfakename luqmanxyz Tradingstrategycourses marcel FiditeNemini Dlbk clem netapy Isotonic satyamt merve dvilasuero Epiculous lunarflu CultriX antiven0m rvpierre mishig Locutusque Citaman Humus macadeliccc ivanfioravanti julien-c acertainbru sugatoray ezzaldeen ahmedelsayed

Dlbk osanseviero mrfakename FiditeNemini clem merve PrimeD Epiculous lunarflu rvpierre Q-bert julien-c kgourgou wsuff nbeerbower

hbzhangvt samusenps osanseviero clem Isotonic merve HuggyMonkey Epiculous lunarflu rvpierre


GPU Poor POV: Low Hanging Fruits

Sometimes we had to work with different language than English (what a surprise!) and it can be problematic, because as you may know many algorithms are mainly developed in English.
I was involved in building RAG in Polish language. At first, we need an proper embeddings for Polish language to feed them into lightweight LLM.
Looking through possible solution I become aware that existing/possible models are not accurate enough, and worked much worse than its 'english equivalent'.
First thing that comes to mind is:
```
 Lets become a mad scientist, download all possible data and train model for months to get the proper one.
```
But there are few cons of this.
- Its computionally heavy
- You are not full time researcher
- you have potential clients who want to use your solution, and they really happy to use it (in optimistic mood).
Here comes the low hanging fruits.
 We developed a easier, workable solution. Instead of training new SOTA, we can use translation module like this one:

https://huggingface.co/Helsinki-NLP/opus-mt-pl-en
translate your knowledge base to english, and use proper embedding model accurately.
I converted existing model using ctranslate2,

```
ct2-transformers-converter --model Helsinki-NLP/opus-mt-pl-en --output_dir opus-mt-pl-en
```
so making an inference  is not heavy (we observe 5 times speedup in compare to original version).

And by indexing knowledge base, we can return answer to LLM  in any language. (Indexes of context found in english language are equal to indexes in native language knowledge base).

Of course there are some tweaks required, we have to validate accuracy of the translation.

It was nice episode, we have our work done, there are people who can use it, so additive value exists.
Have a great day and I wish you more effective deploys! <3

mlabonne Tonic victor Dlbk clem NeuralNovel Ji-Ha

afrideva MaziyarPanahi osanseviero clem saudi-agi


Explaining the 👑 of zero-shot open-vocabulary object detection: OWLv2 🦉
OWLv2 is scaled version of a model called OWL-ViT, so let's take a look at that first. 📝
OWLViT is an open vocabulary object detector, meaning, it can detect objects it didn't explicitly see during the training. 👀
What's cool is that it can take both image and text queries! This is thanks to how the image and text features aren't fused together.

Taking a look at the architecture, the authors firstly do contrastive pre-training of a vision and a text encoder (just like CLIP). They take that model, remove the final pooling layer and attach a lightweight classification and box detection head and fine-tune.
During fine-tuning for object detection, they calculate the loss over bipartite matches. Simply put, loss is calculated over the predicted objects against ground truth objects and the goal is to find a perfect match of these two sets where each object is matched to one object in ground truth.

OWL-ViT is very scalable. You can easily scale most language models or vision-language models because they require no supervision, but this isn't the case for object detection: you still need weak supervision. Moreover, only scaling the encoders creates a bottleneck after a while.

The authors wanted to scale OWL-ViT with more data, so they used OWL-ViT for labelling to train a better detector, "self-train" a new detector on the labels, and fine-tune the model on human-annotated data.

Thanks to this, OWLv2 scaled very well and topped leaderboards on open vocabulary object detection 👑
If you'd like to try it out, I will leave couple of links with apps, notebooks and more in the comments! 🤗

s3nh NeuralNovel joytafty osanseviero not-lain samusenps Skier8402 Erland mlabonne Dlbk taufiqdp Jofthomas clem rreed aidystark johko ayhaidar ceofast


roseking Chrissss Tonic mrfakename NeuralNovel KyonBS radames hayden-donnelly Duskfallcrew DamarJati AMIAUTHOR fumjtea6fg Dlbk afrideva Jofthomas osanseviero umuthopeyildirim FloRul clem den0620 thr3a MarinaraSpaghetti kramp plmsmile mvaloatto fullpwerr abidlabs lunarflu julien-c MoritzLaurer AdinaY mishig TuringsSolutions victor not-lain


Can Large Language Models Explain Themselves? (2401.07927)

🔍 Today's pick in Interpretability & Analysis of LMs: Can Large Language Models Explain Themselves? by @andreasmadsen Sarath Chandar & @sivareddyg

LLMs can provide wrong but convincing explanations for their behavior, and this might lead to ill-placed confidence in their predictions. This study uses self-consistency checks to measure the faithfulness of LLM explanations: if an LLM says a set of words is important for making a prediction, then it should not be able to make the same prediction without these words. Results demonstrate that LLM self-explanations faithfulness of self-explanations cannot be reliably trusted, as they prove to be very task and model dependent, with bigger model generally producing more faithful explanations.

📄 Paper: https://huggingface.co/papers/2401.07927

sivareddyg roseking osanseviero Tonic KnutJaegersberg victor samusenps mlabonne Dlbk clem clefourrier abidlabs


Self-Rewarding Language Models (2401.10020)

Self-Rewarding Language Models

paper page: https://huggingface.co/papers/2401.10020

Fine-tuning Llama 2 70B on three iterations of our approach yields a model that outperforms many existing systems on the AlpacaEval 2.0 leaderboard, including Claude 2, Gemini Pro, and GPT-4 0613

roseking gsarti NeuralNovel Araeynn TheLastBen samusenps Kernel AIIAR eliebak mlabonne bunnycore Norm Dlbk clem orick96


Last week, we released 🤗 Transformers.js v2.14, which added support for SAM (Segment Anything Model).

This means you can now generate high-quality segmentation masks for objects in a scene, directly in your browser! 🤯

Demo (+ source code): https://huggingface.co/spaces/Xenova/segment-anything-web
Model: https://huggingface.co/Xenova/slimsam-77-uniform

But how does this differ from Meta's original demo? 🤔 Didn't that also run in-browser?

Well, in their demo, the image embeddings are computed server-side, then sent to the client for decoding. Trying to do this all client-side would be completely impractical: taking minutes per image! 😵💫

That's where SlimSAM comes to the rescue! SlimSAM is a novel SAM compression method, able to shrink the model over 100x (637M → 5.5M params), while still achieving remarkable results!

The best part? You can get started in a few lines of JavaScript code, thanks to Transformers.js! 🔥

```
// npm i @xenova/transformers
import { SamModel, AutoProcessor, RawImage } from '@xenova/transformers';

// Load model and processor
const model = await SamModel.from_pretrained('Xenova/slimsam-77-uniform');
const processor = await AutoProcessor.from_pretrained('Xenova/slimsam-77-uniform');

// Prepare image and input points
const img_url = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/corgi.jpg';
const raw_image = await RawImage.read(img_url);
const input_points = [[[340, 250]]];

// Process inputs and perform mask generation
const inputs = await processor(raw_image, input_points);
const outputs = await model(inputs);

// Post-process masks
const masks = await processor.post_process_masks(outputs.pred_masks, inputs.original_sizes, inputs.reshaped_input_sizes);
console.log(masks);

// Visualize the mask
const image = RawImage.fromTensor(masks[0][0].mul(255));
image.save('mask.png');
```

I can't wait to see what you build with it! 🤗

osanseviero radames rahuldshetty victor brunnolou samusenps afrideva Dlbk clem rreed zuozuo AdamSako omaryshchenko NikitaKras chinmayak cggaurav bmullan doublelotus horizon0514

smalltong02 mlabonne Sylvestre clem rreed Norod78 AkindOf


Posting about a very underrated model that tops paperswithcode across different segmentation benchmarks: OneFormer 👑

OneFormer is a "truly universal" model for semantic, instance and panoptic segmentation tasks ⚔
What makes is truly universal is that it's a single model that is trained only once and can be used across all tasks.
The enabler here is the text conditioning, i.e. the model is given a text query that states task type along with the appropriate input, and using contrastive loss, the model learns the difference between different task types 👇 (see in the image below)

It's also super easy to use with transformers.
```python
from transformers import OneFormerProcessor, OneFormerForUniversalSegmentation

processor = OneFormerProcessor.from_pretrained("shi-labs/oneformer_ade20k_swin_large")
model = OneFormerForUniversalSegmentation.from_pretrained("shi-labs/oneformer_ade20k_swin_large")

# swap the postprocessing and task_inputs for different types of segmentation
semantic_inputs = processor(images=image, task_inputs=["semantic"], return_tensors="pt")
semantic_outputs = model(**semantic_inputs)
predicted_semantic_map = processor.post_process_semantic_segmentation(outputs, target_sizes=[image.size[::-1]])[0]
```
I have drafted a notebook for you to try right away ✨ https://colab.research.google.com/drive/1wfJhoTFqUqcTAYAOUc6TXUubBTmOYaVa?usp=sharing
You can also check out the Space without checking out the code itself 👉 https://huggingface.co/spaces/shi-labs/OneFormer

gsarti rwightman Chuanming Dlbk osanseviero kingabzpro jxtan clem julien-c victor NeuralNovel mlabonne lunarflu

osanseviero samusenps erfanzar clem julien-c Timar Dlbk vztay lunarflu afrideva

osanseviero clem julien-c freinold Dlbk rreed lunarflu noobmldude


Vision Mamba: Efficient Visual Representation Learning with
  Bidirectional State Space Model (2401.09417)

Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model

paper page: https://huggingface.co/papers/2401.09417

Recently the state space models (SSMs) with efficient hardware-aware designs, i.e., Mamba, have shown great potential for long sequence modeling. Building efficient and generic vision backbones purely upon SSMs is an appealing direction. However, representing visual data is challenging for SSMs due to the position-sensitivity of visual data and the requirement of global context for visual understanding. In this paper, we show that the reliance of visual representation learning on self-attention is not necessary and propose a new generic vision backbone with bidirectional Mamba blocks (Vim), which marks the image sequences with position embeddings and compresses the visual representation with bidirectional state space models. On ImageNet classification, COCO object detection, and ADE20k semantic segmentation tasks, Vim achieves higher performance compared to well-established vision transformers like DeiT, while also demonstrating significantly improved computation & memory efficiency. For example, Vim is 2.8times faster than DeiT and saves 86.8% GPU memory when performing batch inference to extract features on images with a resolution of 1248times1248. The results demonstrate that Vim is capable of overcoming the computation & memory constraints on performing Transformer-style understanding for high-resolution images and it has great potential to become the next-generation backbone for vision foundation models.

clem Goekdeniz-Guelmez osanseviero merve victor samusenps erfanzar Dlbk TheLastBen mlabonne


Most upvoted papers of 2023 on HF. What do you think are going to be the most prominent research topics in AI for 2024 (also, don't forget to add your papers to the hub this year!).

From: https://huggingface.co/spaces/hysts/daily-papers

pierrci osanseviero vwxyzjn mvaloatto linoyts gsarti victor samusenps julien-c Dlbk yukiarimo mlabonne MaziyarPanahi Jofthomas dvilasuero snarktank hysts mfarajtabar avinash02


Patchscope: A Unifying Framework for Inspecting Hidden Representations
  of Language Models (2401.06102)

💥 Today's pick in Interpretability & Analysis of LMs: 🩺 Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models by @asmadotgh, @codevan, @1wheel, @iislucas & @mega

Patchscopes is a generalized framework for verbalizing information contained in LM representations. This is achieved via a mid-forward patching operation inserting the information into an ad-hoc prompt aimed at eliciting model knowledge. Patchscope instances for vocabulary projection, feature extraction and entity resolution in model representation are show to outperform popular interpretability approaches, often resulting in more robust and expressive information.

🌐 Website: https://pair-code.github.io/interpretability/patchscopes/
📄 Paper: https://huggingface.co/papers/2401.06102

s3nh osanseviero victor codevan iislucas asmadotgh merve Kernel samusenps anjutv clem Dlbk


1) 👩🏾💻 𝐃𝐞𝐜𝐢𝐂𝐨𝐝𝐞𝐫-𝟔𝐁

1) 👩🏾💻 𝐃𝐞𝐜𝐢𝐂𝐨𝐝𝐞𝐫-𝟔𝐁

💻 𝑻𝒓𝒚 𝒊𝒕 𝒐𝒖𝒕:

💻 𝑻𝒓𝒚 𝒊𝒕 𝒐𝒖𝒕:

🃏 𝐌𝐨𝐝𝐞𝐥 𝐂𝐚𝐫𝐝:

🃏 𝐌𝐨𝐝𝐞𝐥 𝐂𝐚𝐫𝐝:

🪧 𝐇𝐮𝐠𝐠𝐢𝐧𝐠𝐅𝐚𝐜𝐞 𝐒𝐩𝐚𝐜𝐞:

🪧 𝐇𝐮𝐠𝐠𝐢𝐧𝐠𝐅𝐚𝐜𝐞 𝐒𝐩𝐚𝐜𝐞:

2) 🎨 𝐃𝐞𝐜𝐢𝐃𝐢𝐟𝐟𝐮𝐬𝐢𝐨𝐧 𝐯𝟐.𝟎

2) 🎨 𝐃𝐞𝐜𝐢𝐃𝐢𝐟𝐟𝐮𝐬𝐢𝐨𝐧 𝐯𝟐.𝟎

💻 𝑻𝒓𝒚 𝒊𝒕 𝒐𝒖𝒕:

💻 𝑻𝒓𝒚 𝒊𝒕 𝒐𝒖𝒕:

🃏 𝐌𝐨𝐝𝐞𝐥 𝐂𝐚𝐫𝐝:

🃏 𝐌𝐨𝐝𝐞𝐥 𝐂𝐚𝐫𝐝:

🪧 𝐇𝐮𝐠𝐠𝐢𝐧𝐠𝐅𝐚𝐜𝐞 𝐒𝐩𝐚𝐜𝐞:

🪧 𝐇𝐮𝐠𝐠𝐢𝐧𝐠𝐅𝐚𝐜𝐞 𝐒𝐩𝐚𝐜𝐞:

✌🏼Two new models dropped today 👇🏽

1) 👩🏾💻 𝐃𝐞𝐜𝐢𝐂𝐨𝐝𝐞𝐫-𝟔𝐁

👉🏽 Supports 𝟖 𝐥𝐚𝐧𝐠𝐮𝐚𝐠𝐞𝐬: C, C# C++, GO, Rust, Python, Java, and Javascript.

👉🏽 Released under the 𝐀𝐩𝐚𝐜𝐡𝐞 𝟐.𝟎 𝐥𝐢𝐜𝐞𝐧𝐬𝐞

🥊 𝐏𝐮𝐧𝐜𝐡𝐞𝐬 𝐚𝐛𝐨𝐯𝐞 𝐢𝐭𝐬 𝐰𝐞𝐢𝐠𝐡𝐭 𝐜𝐥𝐚𝐬𝐬 𝐨𝐧 𝐇𝐮𝐦𝐚𝐧𝐄𝐯𝐚𝐥: Beats out CodeGen 2.5 7B and StarCoder 7B on most supported languages. Has a 3-point lead over StarCoderBase 15.5B for Python

💻 𝑻𝒓𝒚 𝒊𝒕 𝒐𝒖𝒕:

🃏 𝐌𝐨𝐝𝐞𝐥 𝐂𝐚𝐫𝐝: https://huggingface.co/Deci/DeciCoder-6B

📓 𝐍𝐨𝐭𝐞𝐛𝐨𝐨𝐤: https://colab.research.google.com/drive/1QRbuser0rfUiFmQbesQJLXVtBYZOlKpB

🪧 𝐇𝐮𝐠𝐠𝐢𝐧𝐠𝐅𝐚𝐜𝐞 𝐒𝐩𝐚𝐜𝐞: https://huggingface.co/spaces/Deci/DeciCoder-6B-Demo

2) 🎨 𝐃𝐞𝐜𝐢𝐃𝐢𝐟𝐟𝐮𝐬𝐢𝐨𝐧 𝐯𝟐.𝟎

👉🏽 Produces quality images on par with Stable Diffusion v1.5, but 𝟐.𝟔 𝐭𝐢𝐦𝐞𝐬 𝐟𝐚𝐬𝐭𝐞𝐫 𝐢𝐧 𝟒𝟎% 𝐟𝐞𝐰𝐞𝐫 𝐢𝐭𝐞𝐫𝐚𝐭𝐢𝐨𝐧𝐬

👉🏽  Employs a 𝐬𝐦𝐚𝐥𝐥𝐞𝐫 𝐚𝐧𝐝 𝐟𝐚𝐬𝐭𝐞𝐫 𝐔-𝐍𝐞𝐭 𝐜𝐨𝐦𝐩𝐨𝐧𝐞𝐧𝐭 𝐰𝐡𝐢𝐜𝐡 𝐡𝐚𝐬 𝟖𝟔𝟎 𝐦𝐢𝐥𝐥𝐢𝐨𝐧 𝐩𝐚𝐫𝐚𝐦𝐞𝐭𝐞𝐫𝐬.

👉🏽  Uses an optimized scheduler, 𝐒𝐪𝐮𝐞𝐞𝐳𝐞𝐝𝐃𝐏𝐌++, which 𝐜𝐮𝐭𝐬 𝐝𝐨𝐰𝐧 𝐭𝐡𝐞 𝐧𝐮𝐦𝐛𝐞𝐫 𝐨𝐟 𝐬𝐭𝐞𝐩𝐬 𝐧𝐞𝐞𝐝𝐞𝐝 𝐭𝐨 𝐠𝐞𝐧𝐞𝐫𝐚𝐭𝐞 𝐚 𝐪𝐮𝐚𝐥𝐢𝐭𝐲 𝐢𝐦𝐚𝐠𝐞 𝐟𝐫𝐨𝐦 𝟏𝟔 𝐭𝐨 𝟏𝟎.

👉🏽 Released under the 𝐂𝐫𝐞𝐚𝐭𝐢𝐯𝐞𝐌𝐋 𝐎𝐩𝐞𝐧 𝐑𝐀𝐈𝐋++-𝐌 𝐋𝐢𝐜𝐞𝐧𝐬𝐞.

💻 𝑻𝒓𝒚 𝒊𝒕 𝒐𝒖𝒕:

🃏 𝐌𝐨𝐝𝐞𝐥 𝐂𝐚𝐫𝐝: https://huggingface.co/Deci/DeciDiffusion-v2-0

📓 𝐍𝐨𝐭𝐞𝐛𝐨𝐨𝐤: https://colab.research.google.com/drive/11Ui_KRtK2DkLHLrW0aa11MiDciW4dTuB

🪧 𝐇𝐮𝐠𝐠𝐢𝐧𝐠𝐅𝐚𝐜𝐞 𝐒𝐩𝐚𝐜𝐞: https://huggingface.co/spaces/Deci/DeciDiffusion-v2-0

Help support the projects by liking the model cards and the spaces!

Cheers and happy hacking!

osanseviero merve davanstrien cabralski Quinn-03 kramp itsn78 ailaughface clem alielfilali01 lunarflu Tanvir1337 Dlbk NeuralNovel joytafty VictorSanh den0620 Tonic

osanseviero merve akashicmarga ailaughface clem anjutv ysharma linoyts lunarflu Tanvir1337 Dlbk joytafty samusenps sbrandeis


𝐀𝐧 𝐈𝐝𝐞𝐚 𝐟𝐫𝐨𝐦 𝐭𝐡𝐞 𝐇𝐞𝐚𝐫𝐭

𝐀𝐧 𝐈𝐝𝐞𝐚 𝐟𝐫𝐨𝐦 𝐭𝐡𝐞 𝐇𝐞𝐚𝐫𝐭

𝐄𝐦𝐛𝐫𝐚𝐜𝐞𝐝 𝐛𝐲 𝐇𝐮𝐠𝐠𝐢𝐧𝐠 𝐅𝐚𝐜𝐞: 𝐭𝐡𝐞 𝐈𝐧𝐬𝐢𝐝𝐞 𝐒𝐭𝐨𝐫𝐲 𝐨𝐟 𝐎𝐮𝐫 𝐒𝐭𝐚𝐫𝐭𝐮𝐩’𝐬 𝐀𝐜𝐪𝐮𝐢𝐬𝐢𝐭𝐢𝐨𝐧

In late 2021, our team of five engineers, scattered around the globe, signed the papers to shut down our startup, Gradio. For many founders, this would have been a moment of sadness or even bitter reflection.

But we were celebrating. We were getting acquired by Hugging Face!

We had been working very hard towards this acquisition, but for weeks, the acquisition had been blocked by a single investor. The more we pressed him, the more he buckled down, refusing to sign off on the acquisition. Until, unexpectedly, the investor conceded, allowing us to join Hugging Face.

For the first time since our acquisition, I’m writing down the story in detail, hoping that it may shed some light into the obscure world of startup acquisitions and what decisions founders can make to improve their odds for a successful acquisition.

To understand how we got acquired by Hugging Face, you need to know why we started Gradio.

𝐀𝐧 𝐈𝐝𝐞𝐚 𝐟𝐫𝐨𝐦 𝐭𝐡𝐞 𝐇𝐞𝐚𝐫𝐭

Two years before the acquisition, in early 2019, I was working on a research project at Stanford. It was the third year of my PhD, and my labmates and I had trained a machine learning model that could predict patient biomarkers (such as whether patients had certain diseases or an implanted pacemaker) from an ultrasound image of their heart — as well as a cardiologist.

Naturally, cardiologists were skeptical... read the rest of the story here: https://twitter.com/abidlabs/status/1745533306492588303

osanseviero BrigitteTousi EmilyWitko jeffboudier samusenps harpreetsahota tomaarsen alielfilali01 radames gsarti vinidlidoo pierrci Tonic chansung DopeorNope woolr clem Sd1p ysharma lunarflu susnato rwightman dvilasuero

BrigitteTousi samusenps harpreetsahota tomaarsen gsarti Tonic chansung woolr sbarman25 clem Sd1p kramp lunarflu


🪟32k-context BERT for embedding and RAG on long corpus

Monarch Mixer is a new architecture to enable long context BERT for large corpus and can be fine-tuned for large context retrieval.

Quite interesting and important as BERT is still the most used LLM in production for "old school" tasks like classification, NER, embeddings, but is also a key component for RAG.

Paper: https://arxiv.org/abs/2310.12109
Blog: https://hazyresearch.stanford.edu/blog/2024-01-11-m2-bert-retrieval
GitHub: https://github.com/HazyResearch/m2


Scalable Pre-training of Large Autoregressive Image Models (2401.08541)

Scalable Pre-training of Large Autoregressive Image Models

paper page: https://huggingface.co/papers/2401.08541

paper introduces AIM, a collection of vision models pre-trained with an autoregressive objective. These models are inspired by their textual counterparts, i.e., Large Language Models (LLMs), and exhibit similar scaling properties. Specifically, we highlight two key findings: (1) the performance of the visual features scale with both the model capacity and the quantity of data, (2) the value of the objective function correlates with the performance of the model on downstream tasks. We illustrate the practical implication of these findings by pre-training a 7 billion parameter AIM on 2 billion images, that achieves 84.0% on ImageNet-1k with a frozen trunk. Interestingly, even at this scale, we observe no sign of saturation in performance, suggesting that AIM potentially represents a new frontier for training large-scale vision models. The pre-training of AIM is similar to the pre-training of LLMs, and does not require any image-specific strategy to stabilize the training at scale.

gsarti taufiqdp clem rwightman NeuralNovel


Up to 3x faster LLM generation with no extra resources/requirements - ngram speculation has landed in 🤗 transformers! 🏎💨

All you need to do is to add `prompt_lookup_num_tokens=10` to your `generate` call, and you'll get faster LLMs 🔥

How does it work? 🤔

Start with assisted generation, where a smaller model generates candidate sequences. The net result is a significant speedup if the model agrees with the candidate sequences! However, we do require a smaller model trained similarly 😕

The idea introduced (and implemented) by Apoorv Saxena consists of gathering the candidate sequences from the input text itself. If the latest generated ngram is in the input, use the continuation therein as a candidate! No smaller model is required while still achieving significant speedups 🔥

In fact, the penalty of gathering and testing the candidates is so small that you should use this technique whenever possible!

Here is the code example that produces the outputs shown in the video: https://pastebin.com/bms6XtR4

Have fun 🤗

samusenps Dlbk akashicmarga nickandbro osanseviero abidlabs clem alielfilali01 gblazex DamarJati jaigouk Goekdeniz-Guelmez MaziyarPanahi nbroad abdullah m-ric clefourrier takeraparterer Felladrin

samusenps osanseviero clem alielfilali01 Svngoku takeraparterer

MoritzLaurer osanseviero DRXD1000 manu clem takeraparterer


Real-Time Vehicle Speed Estimation Tutorial 🚗💨💨💨

TL;DR: Watch the tutorial here: https://www.youtube.com/watch?v=uWP6UjDeZvY

Key Steps:
1. Vehicle Detection: Before we jump into speed estimation, we begin by detecting moving vehicles. I demonstrate this using YOLOv8, deployed through the Inference pip package.

2. Tracking with ByteTrack: For effective object tracking, ByteTrack is my tool of choice. It assigns a unique ID to each vehicle, which is essential for accurately monitoring the distance each car travels. This forms the cornerstone of our speed calculation process.

3. Distance Calculation Complexities: Calculating traveled distance can be tricky due to perspective distortion from the camera. A car moving at a constant speed will appear to move a different number of pixels in the image, depending on its distance from the camera.

4. Vehicle Positioning: We can accurately pinpoint each vehicle's position within our monitored area. By representing each vehicle with x and y coordinates in meters, we can compare its current and past positions, paving the way for calculating its speed.

5. We store the position of each car in the last second, calculate the offset, and divide it by the time delta to get the local speed.

- 🔗 tutorial: https://www.youtube.com/watch?v=uWP6UjDeZvY
- 🔗 code: https://github.com/roboflow/supervision/tree/develop/examples/speed_estimation

SkalskiP merve osanseviero julien-c HaohuaLv kramp muhtasham samusenps yeldarb mpfmorawski radames clem jxtan vince62s lynguyenminh victor

osanseviero samusenps radames clem SkalskiP mrfakename


if you're encountering 500 errors on spaces that seem to work otherwise , kindly consider screenshotting and sharing the link here : https://discord.com/channels/879548962464493619/1295847667515129877


🚨🔥 New Release Alert! 🔥🚨

Introducing the 435M model that outperforms Llama-Guard-3-8B while slashing 75% of the computation cost! 💻💥
👉 Check it out: https://huggingface.co/hbseong/HarmAug-Guard (Yes, INFERENCE CODE INCLUDED! 💡)

More details in our paper: https://arxiv.org/abs/2410.01524 📜

#HarmAug #LLM # Safety #EfficiencyBoost #Research #AI #MachineLearning

Seanie-lee hbseong John6666 adorkin AtAndDev

FM-1976 AtAndDev TuringsSolutions hbseong


🚀 Child Safe Chatbot (Experimental)Based on google/shieldgemma-9b. Currently in the experimental phase, this configuration could be a starting point to consolidate an LLM that meets the requirements to be safe for minors.
👉 https://huggingface.co/spaces/nmarafo/Child-Safe-Chatbot
#EdTech #ChildSafety #AI #Education #Experimental


🚀 Introducing ColFlor: An Efficient, OCR-Free Vision-Language Document Retrieval Model 🌟

Earlier this year, ColPali revolutionized document retrieval by eliminating the need for error-prone OCR pipelines. Instead, it directly processes the document images. However, with its 3 billion parameters, ColPali is computationally heavy for large-scale applications.

That’s where ColFlor comes in—a smaller, faster alternative! 🎉 At 17x smaller than ColPali, ColFlor offers a more efficient, OCR-free document retrieval solution, making it ideal for users with limited computing resources (GPU Poor). 💡

Key Highlights:
🧠 174M parameters (vs. 3B for ColPali)
⚡ 9.8x faster query encoding, 5.25x faster image encoding
📉 Only 1.8% performance drop on text-rich English documents

Check out the full blog post for more insights on modeling, training, and evaluations across various document retrieval tasks! 🚀
Also, feel free to try our demo on huggingface 🤗

🔗 Resources:
📄 Blog post: https://huggingface.co/blog/ahmed-masry/colflor
🧠 Model: https://huggingface.co/ahmed-masry/ColFlor
💻 Demo: https://huggingface.co/spaces/ahmed-masry/ColFlor-Demo
🏋♂ Training code: https://github.com/AhmedMasryKU/colflor
📊 Evaluation code: https://github.com/AhmedMasryKU/vidore-benchmark-colflor


I built a Hyper Dimensional Computing (HDC) Encoder and Decoder and I also built out all of the code needed to replace the Encoder and Decoder of a Llama model with this HDC model, then train the Llama model on the HDC Encoder/Decoder. All MIT licensed. Here is a video where I break it all down. I can answer any questions about this project or help anyone out where I can. I am not a super developer or anything and I don't have access to enough compute to train this on a large dataset: https://youtu.be/4VsZpGaPK4g


Here is how we can calculate the size of any LLM model:

Each parameter in LLM models is typically stored as a floating-point number. The size of each parameter in bytes depends on the precision.

32-bit precision: Each parameter takes 4 bytes.
16-bit precision: Each parameter takes 2 bytes

To calculate the total memory usage of the model:
Memory usage (in bytes) = No. of Parameters × Size of Each Parameter

For example:
32-bit Precision (FP32)
In 32-bit floating-point precision, each parameter takes 4 bytes.
Memory usage in bytes = 1 billion parameters × 4 bytes
1,000,000,000 × 4 = 4,000,000,000 bytes
In gigabytes: ≈ 3.73 GB

16-bit Precision (FP16)
In 16-bit floating-point precision, each parameter takes 2 bytes.
Memory usage in bytes = 1 billion parameters × 2 bytes
1,000,000,000 × 2 = 2,000,000,000 bytes
In gigabytes: ≈ 1.86 GB

It depends on whether you use 32-bit or 16-bit precision, a model with 1 billion parameters would use approximately 3.73 GB or 1.86 GB of memory, respectively.

zzlxv John6666 danielus Ninjanimus san122 SporkySporkness AtAndDev adriszmar


🎓 Introducing Ukr-lit.com.ua Presentations Dataset - https://huggingface.co/datasets/nyuuzyou/ukr-lit

Dataset highlights:
- 18,001 presentations from ukr-lit.com.ua, a platform for storing and viewing presentations covering a wide range of subjects in Ukrainian school education
- Primarily in Ukrainian, with some Russian and English content
- Each entry includes: URL, title, download URL, filepath, and extracted text content (where available)
- Contains original PPT/PPTX files in addition to metadata
- Data covers a broad spectrum of educational topics and subjects taught in Ukrainian schools
- Dedicated to the public domain under Creative Commons Zero (CC0) license

The dataset can be used for analyzing educational presentation content across various subjects in Ukrainian and other languages, text classification tasks, and information retrieval systems. It's particularly valuable for examining trends in Ukrainian school education, teaching methodologies, and presentation materials used across different academic disciplines. The inclusion of original files allows for in-depth analysis of presentation formats and structures commonly used in Ukrainian educational settings, providing insights into the diverse range of subjects and teaching approaches in the Ukrainian school system.


🌍 Super cool visualization of global PUT requests to Hugging Face over 24 hours, coded by object size, thanks to @port8080!

We're putting this analysis to work to help us architect a more geo-distributed system for the HF storage backend.

Originally shared on LinkedIn: https://www.linkedin.com/posts/ajitbanerjee_one-of-the-joys-of-working-on-the-xethub-activity-7252688424732614656-tFGD

nyuuzyou port8080 Awhildy AtAndDev John6666 gabrielmbmb not-lain yjernite xi0v jsulz


Big news! You can now build strong ML models without days of human labelling

You simply:
- Define your dataset, including annotation guidelines, labels and fields
- Optionally label some records manually.
- Use an LLM to auto label your data with a human (you? your team?) in the loop!

Get started with this blog post:
https://huggingface.co/blog/sdiazlor/custom-text-classifier-ai-human-feedback


You can now build a custom text classifier without days of human labeling!

👍 LLMs work reasonably well as text classifiers.
👎 They are expensive to run at scale and their performance drops in specialized domains.

👍 Purpose-built classifiers have low latency and can potentially run on CPU.
👎 They require labeled training data.

Combine the best of both worlds: the automatic labeling capabilities of LLMs and the high-quality annotations from human experts to train and deploy a specialized model.

Blog: https://huggingface.co/blog/sdiazlor/custom-text-classifier-ai-human-feedback

dvilasuero AtAndDev NickoSELI tiendung Darklight0012 davidberenstein1957 clem

nyuuzyou John6666 victor Ninjanimus AtAndDev tiendung davidberenstein1957


Fine-grained Hallucination Detection and Editing for Language Models (2401.06855)

💥 Today's pick in Interpretability & Analysis of LMs: Fine-grained Hallucination Detection and Editing For Language Models by @abhika-m @akariasai @vidhisha et al.

Authors introduce a new taxonomy for fine-grained annotation of hallucinations in LM generations and propose Factuality Verification with Augmented Knowledge (FAVA), a retrieval-augmented LM fine-tuned to detect and edit hallucinations in LM outputs, outperforming ChatGPT and LLama2 Chat on both detection and editing tasks.

🌐 Website: https://fine-grained-hallucination.github.io
📄 Paper: https://huggingface.co/papers/2401.06855
🚀 Demo: https://huggingface.co/spaces/fava-uw/fava
🤖 Model: https://huggingface.co/fava-uw/fava-model
🔡 Dataset: https://huggingface.co/datasets/fava-uw/fava-data

merve osanseviero samusenps abhika-m yihaopeng d0rj clem julien-c joytafty abdullah dhuynh95 rreed sbrandeis

samusenps abhika-m clem julien-c joytafty rreed


GPU Poor POV: Building a RAG which solves specific task.

Everyone loves benchmarks.
They are great because we have standarized approach,  competitive feeling. But if you are in specific area, trying  to implement some LLM/RAG use case, these benchmarks  cannot exactly reflect on the data that you have to deal with.

I built RAG system on bunch of niche procedures/regulation etc,  which can be finally deployed as an virtual assistant to minimize the effort in searching through a lot of documentations manually.

Tested a lot of different methods/models/pretrains, finetunes and whats interesting is that, final solution which was scored by human feedback is based on relatively low param models, with multitask ability
Something like:

https://huggingface.co/BAAI/llm-embedder

LLMs help summarize the chunk version of knowledge base found, does not require the model with high number of params, because tradeoff between inference time and accuracy has to be made. Some lightweight models have ability to perform certain task based on instructions,  so eg. qwen 7b or mistral 7b (not moe one), realized a task really nicely. And what is more important is that in overall we are able to deploy a RAG system in smaller tasks, in specific area. They can be used by people who need it, give additive value and positive feedback, which IMO is what is all of the building process about.

Have a great day and think about problem which your models have to solve <3

100rab RASMUS victor merve osanseviero tomaarsen samusenps clem Citaman lunarflu Tanvir1337 sethuiyer arpitingle Ji-Ha

merve samusenps akashicmarga clem lunarflu TuringsSolutions Tanvir1337 sbrandeis

victor merve clem lunarflu TuringsSolutions Tanvir1337 s3nh


What is the current SOTA in terms of fast personalized image generation? Most of the techniques that produce great results (which is hard to objectively measure, but subject similarity index being close to 80-90%) take either too much time (full on DreamBooth fine-tuning the base model) or or loose on the auxilary properties (high rank LoRAs).

We have been also testing face embeddings, but even with multiple samples the quality is not anywhere close to what we expect. Even on the techniques that work, high quality (studio-level) pictures seem to be a must so another avenue that I'm curious is whether there is filter/segmentation of the input samples in an automatic way that people have looked in the past?

osanseviero chansung samusenps victor merve clem

samusenps akashicmarga clem badayvedat quantumcomputer


Google's SigLIP is another alternative to openai's CLIP, and it  just got merged to 🤗transformers and it's super easy to use!
To celebrate this, I have created a repository including notebooks and bunch of Spaces on various SigLIP based projects 🥳
Search for art 👉 https://huggingface.co/spaces/merve/draw_to_search_art
Compare SigLIP with CLIP 👉 https://huggingface.co/spaces/merve/compare_clip_siglip

How does SigLIP work?
SigLIP an vision-text pre-training technique based on contrastive learning. It jointly trains an image encoder and text encoder such that the dot product of embeddings are most similar for the appropriate text-image pairs
The image below is taken from CLIP, where this contrastive pre-training takes place with softmax, but SigLIP replaces softmax with sigmoid. 📎

Highlights from the paper on why you should use it ✨
🖼📝 Authors used medium sized B/16 ViT for image encoder and B-sized transformer for text encoder
😍 More performant than CLIP on zero-shot
🗣 Authors trained a multilingual model too!
⚡ Super efficient, sigmoid is enabling up to 1M items per batch, but the authors chose 32k because the performance saturates after that

It's super easy to use thanks to transformers 👇
```python
from transformers import pipeline
from PIL import Image
import requests

# load pipe
image_classifier = pipeline(task="zero-shot-image-classification", model="google/siglip-base-patch16-256-i18n")

# load image
url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)

# inference
outputs = image_classifier(image, candidate_labels=["2 cats", "a plane", "a remote"])
outputs = [{"score": round(output["score"], 4), "label": output["label"] } for output in outputs]
print(outputs)
```
For all the SigLIP notebooks on similarity search and indexing, you can check this [repository](https://github.com/merveenoyan/siglip) out. 🤗

dreamworks2050 victor dvilasuero julien-c osanseviero samusenps Leyo abidlabs TuringsSolutions chansung merve Kernel cdr6934 clem cataluna84

samusenps TuringsSolutions pryanshusharma rwightman chansung linoyts SRDdev akashicmarga yihaopeng clem afrideva cataluna84

nezubn VictorSanh dvilasuero julien-c osanseviero TuringsSolutions chansung clem cataluna84


Is synthetic data the future of AI? 🔥🔥🔥

@HugoLaurencon @Leyo & @VictorSanh are introducing https://huggingface.co/datasets/HuggingFaceM4/WebSight , a multimodal dataset featuring 823,000 pairs of synthetically generated HTML/CSS codes along with screenshots of the corresponding rendered websites to train GPT4-V-like models 🌐💻

While crafting their upcoming foundation vision language model, they faced the challenge of converting website screenshots into usable HTML/CSS codes. Most VLMs suck at this and there was no public dataset available for this specific task, so they decided to create their own.

They prompted existing LLMs to generate 823k HTML/CSS codes of very simple websites. Through supervised fine-tuning of a vision language model on WebSight, they were able to generate the code to reproduce a website component, given a screenshot.

You can explore the dataset here: https://huggingface.co/datasets/HuggingFaceM4/WebSight

What do you think?

Leyo vishaal27 lhoestq AdinaY merve Hotsnow s3nh florentgbelidji HugoLaurencon cabralski ronzyponz mishig HugoDzz victor chansung kramp ameerazam08 wjbmattingly mcpotato taufiqdp osanseviero phanes alielfilali01 duxans aliabd taesiri pierrci lvwerra VictorSanh Triyansha mrfakename dvilasuero Srulikbd julien-c samusenps Chinar PeepDaSlan9 nic221 DEKUBE LouMa Mithilss rohanbalkondekar SaylorTwift peterschmidt85 yihaopeng mkemka radames faust01 lunarflu vztay iblub zeynel rreed

s3nh lhoestq AdinaY BramVanroy merve cabralski davanstrien Pwlot mishig HansKlepitko HugoDzz bstraehle Sylvestre neovalle ameerazam08 wjbmattingly osanseviero aliabd sungkim11 clementperrot cyberandy dvilasuero julien-c samusenps mrdbourke PeepDaSlan9 Frantz103 LouMa Kernel MoritzLaurer yihaopeng lunarflu Kathakali zeynel z11h rreed freddyaboulton

happymanda cabralski mishig ayut HugoDzz clefourrier ameerazam08 osanseviero aliabd dvilasuero julien-c samusenps PeepDaSlan9 DEKUBE rohanbalkondekar MaziyarPanahi yihaopeng lunarflu rreed jeffboudier sbrandeis

taufiqdp cabralski mishig Wauplin holymonkey47 aliabd doxiy dreamworks2050 Srulikbd julien-c lunarflu binga


Happy to announce, brand new, open-source Hugging Face Competitions platform 🚀 Now, create a machine learning competition for your friends, colleagues or the world for FREE* and host it on Hugging Face: the AI community building the future. Creating a competition requires only two steps: `pip install competitions`, then run `competitions create` and create competition by answering a few questions 💥 Checkout the github repo: https://github.com/huggingface/competitions and docs: https://hf.co/docs/competitions

clem merve robmarkcole linoyts kurianbenoy victor alielfilali01 abhishek AdinaY osanseviero Tonic gsarti nezubn VictorSanh julien-c mrdbourke MaziyarPanahi samusenps Smooke d0rj dillfrescott Ammok ecevinoth PinakHolder

happymanda ThetaPhiPsi chansung victor AdinaY osanseviero doxiy julien-c kramp Smooke Solshine dillfrescott Ammok

mc0ps alielfilali01 pacoid osanseviero julien-c TuringsSolutions Smooke mrfakename Solshine dillfrescott IndiraMandalapu

clem dreamworks2050 julien-c dillfrescott


🙋🏻♂hey there folks,

Everyone's🗣talking about microsoft's new e5mistral embeddings model

🤔🤔 but did you actually try it yet ?

Well , now you can, just check it out. it's a new way to serve and create embeddings.

try it hosted on GPUZero : https://huggingface.co/spaces/Tonic/e5
or served on an A10G : https://huggingface.co/spaces/Tonic/e5

you get best results actually building with it though, so use it in your app !

Our demo is coming soon too so let's work together if you want :-)

kramp s3nh clem osanseviero AdinaY abidlabs eddy8383 nickandbro tomaarsen

clem linoyts samusenps sosoai tomaarsen jilp00


Just published a quick community blog post mainly aimed at Art and Design students, but which is also an attempt to nudge AI researchers who would like to better consider benefits from collaboration with designers and artists 😉
Feel free to share your thoughts !

"Breaking Barriers: The Critical Role of Art and Design in Advancing AI Capabilities" 📄 https://huggingface.co/blog/fffiloni/the-critical-role-of-art-and-design-in-advancing-a

—
This short publication follows the results of two AI Workshops that took place at École des Arts Décoratifs - Paris, lead by Etienne Mineur, Vadim Bernard, Martin de Bie, Antoine Pintout & Sylvain Filoni.

victor lbourdois kramp osanseviero julien-c radames tonyassi chansung Tonic pierrci mishig clem merve linoyts AdinaY not-lain lucasjin sbrandeis

smellsworth clem merve not-lain samusenps OmbelineM


💡We recently launched a Discord server for #Dutch #NLP and #LLMs. We have more than 50 users of _very_ varrying backgrounds! 🧙👩🔬🧑🎨🧑🏫🧑💼 We've already had discussions on eval, tokenizers, RAG, data... A bit of everything. Everyone is welcome to work together on Dutch NLP and LLMs!  https://discord.gg/YUUXVZkZJ9

stefan-it lbourdois osanseviero Tonic gsarti samusenps abidlabs mariagrandury yjernite sbrandeis


As the amount of datasets for fine tuning chat models has grown, there's been a plethora of dataset formats emerge. The most popular of these include the formats used by Alpaca, ShareGPT and Open Assistant datasets. The datasets and their formats have also evolved from single-turn conversation to multi-turn. Many of these formats share similarities (and they all have the same goal), but handling the variations in formats across datasets is often a hassle, and source of potential bugs.

Luckily the community seems to be converging on a simple and elegant chat dataset format: a list with each record being an array with each conversation turn being an object with a role (system, assistant or user) and content. Hugging Face uses this input format in the [Templates for Chat Models](https://huggingface.co/docs/transformers/main/en/chat_templating#how-do-i-use-chat-templates) docs:

```python
messages = [
    {
        "role": "system",
        "content": "You are a friendly chatbot who always responds in the style of a pirate",
    },
    {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
]
```

Popular datasets like https://huggingface.co/datasets/HuggingFaceH4/no_robots follow this format.

To encourage usage of this format, I propose we give it a name: Hugging Face MessagesList format.

The format is defined as:

- Having at least one `messages` column of type list.
- Each messages record is an array containing one or more message turn objects.
- A message turn must have `role` and `content` keys.
- `role` should be one of `system`, `assistant` or `user`.
- `content` is the text content of the message.

This may be a small thing, but having a common dataset format will reduce wasted time data wrangling and help everyone.

davanstrien osanseviero Weyaxi lhoestq julien-c clem merve alielfilali01 andysalerno dvilasuero samusenps afrideva IFSDrake sbrandeis

victor julien-c Tonic dvilasuero TuringsSolutions madroid davanstrien clefourrier