I've tried DoRA (https://arxiv.org/abs/2402.09353) with SDXL using PEFT, outputs are quite detailed 🤩🌟
as usual trained on lego dataset I compiled, I compared them with previously trained pivotal tuned model and the normal DreamBooth model before that 😊

Notebook by @linoyts https://colab.research.google.com/drive/134mt7bCMKtCYyYzETfEGKXT1J6J50ydT?usp=sharing
Integration to PEFT by @BenjaminB https://github.com/huggingface/peft/pull/1474 (more info in the PR)

jeanflop vladbogo NeuralKartMocker davideuler osanseviero Falah harpreetsahota jucamohedano ajibawa-2023 terpjwu1 CarlLee mvaloatto danielhanchen smangrul adamelliotfields fffiloni rafaelpierrehf sgch2023 radames mathiasn1 julien-c

jeanflop Falah danielhanchen smangrul rafaelpierrehf


FuseChat: Knowledge Fusion of Chat Models (2402.16107)

🚀🔥🌟 New Research Alert - ICLR 2024! 🌟🔥🚀
📄 Title: FuseChat: Revolutionizing Chat Models Fusion 🌟🚀

👥 Authors: @Wanfq, @passerqxj et al.

📅 Conference: ICLR, May 7-11, 2024 | Vienna, Austria 🇦🇹

🔗 Paper: https://huggingface.co/papers/2402.16107
🔗 Repository: https://github.com/fanqiwan/FuseLLM

🔥 Models 🤖:
1⃣ FuseChat-7B-VaRM: https://huggingface.co/FuseAI/FuseChat-7B-VaRM
2⃣ FuseChat-7B-Slerp: https://huggingface.co/FuseAI/FuseChat-7B-Slerp
3⃣ OpenChat-3.5-7B-Solar: https://huggingface.co/FuseAI/OpenChat-3.5-7B-Solar
4⃣ FuseChat-7B-TA: https://huggingface.co/FuseAI/FuseChat-7B-TA
5⃣ OpenChat-3.5-7B-Mixtral: https://huggingface.co/FuseAI/OpenChat-3.5-7B-Mixtral

📚 More Papers: more cutting-edge research presented at other conferences in the https://huggingface.co/spaces/DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin

🔍 Keywords: #FuseChat #ChatModels #KnowledgeFusion #ICLR2024 #AI #Innovation #FuseLLM

osanseviero tofuCheng vtiyyal1 Hoioi danielhanchen Concor sbrandeis


Genie: Generative Interactive Environments (2402.15391)

Genie is a new method from Google DeepMind that generates interactive, action-controllable virtual worlds from unlabelled internet videos using.

Keypoints:
* Genie leverages a spatiotemporal video tokenizer, an autoregressive dynamics model, and a latent action model to generate controllable video environments.
* The model is trained on video data alone, without requiring action labels, using unsupervised learning to infer latent actions between frames.
* The method restricts the size of the action vocabulary to 8 to ensure that the number of possible latent actions remains small.
* The dataset used for training is generated by filtering publicly available internet videos with specific criteria related to 2D platformer games for a total of 6.8M videos used for training.

Paper: https://huggingface.co/papers/2402.15391
Project page: https://sites.google.com/view/genie-2024/
More detailed overview in my blog: https://huggingface.co/blog/vladbogo/genie-generative-interactive-environments

Congrats to the authors for their work!

jsunster merve osanseviero kramp terpjwu1 programORdie2 mvaloatto danielhanchen Concor 2dts notune thomwolf julien-c


https://huggingface.co/google/gemma-7b-it is super good!

I wasn't convinced at first, but after vibe-checking it...I'm quite impressed.

I've got a notebook here, which is kind of a framework for vibe-checking LLMs.

In this notebook, I take Gemma for a spin on a variety of prompts:
•  [nonsensical tokens](https://huggingface.co/datasets/harpreetsahota/diverse-token-sampler]
• [conversation where I try to get some PII)(https://huggingface.co/datasets/harpreetsahota/red-team-prompts-questions)
• [summarization ability](https://huggingface.co/datasets/lighteval/summarization)
• [instruction following](https://huggingface.co/datasets/harpreetsahota/Instruction-Following-Evaluation-for-Large-Language-Models]
• [chain of thought reasoning](https://huggingface.co/datasets/ssbuild/alaca_chain-of-thought)

I then used LangChain evaluators (GPT-4 as judge), and track everything in LangSmith. I made public links to the traces where you can inspect the runs.

I hope you find this helpful, and I am certainly open to feedback, criticisms, or ways to improve.

Cheers:

You can find the notebook here: https://colab.research.google.com/drive/1RHzg0FD46kKbiGfTdZw9Fo-DqWzajuoi?usp=sharing

clem mvaloatto lunarflu osanseviero diwank harpreetsahota danielhanchen eramax arjunpatel adamelliotfields sbarman25 pansapiens


Terribly excited about open-source + on-device AI these days! Great to see @qualcomm release 80+ models optimized and curated for their devices and chips on HF: https://huggingface.co/qualcomm

DmitryRyumin vladbogo harpreetsahota adamelliotfields banghua mvaloatto lunarflu osanseviero marceloxp ajibawa-2023 terpjwu1 danielhanchen yjernite samusenps realtamimuddin

lunarflu osanseviero danielus ehristoforu siyengfeng yjernite realtamimuddin


Want more “good machine learning” in your X feed? Here is a new Space for you:
🔔 Top HF Users To Follow On X - https://huggingface.co/spaces/mvaloatto/HF2X

Ever since I fell down the AI rabbit hole, it hasn’t been super easy to spot and follow the most impactful Hugging Face contributors on X. So, inspired by @Weyaxi leaderboards, I decided to create a list just for this purpose.

Why, you ask?

First, it’s quite surprising how so many talented AI pioneers and independent contributors on X don't get the visibility/reach you might expect. Sad but true: follower count doesn't always match up with the value or innovation an individual brings to the table (just stating the obvious here).

Open source AI, in particular, thrives not just on innovation but also on the collective spirit of its believers and builders. With Hugging Face standing out as a prime hub for top AI engineers and contributors, compiling a directory of X profiles from influential figures on this platform felt like a natural step.

This Space aims to not only connect these top contributors but also guide open AI enthusiasts and newcomers towards the field's leading lights.

I put this modest page together using some web scraping and what I remember from my web dev class ages ago! Suggestions/likes are welcome - I’m hoping to keep tweaking/upgrading it, especially if you all find it useful.

Now, let’s follow each other! It’s time to accelerate the dissemination of our ideas, encourage collaboration within our community, and ensure that open AI developments receive the attention and recognition they deserve. 🔥

clem DmitryRyumin osanseviero Alexandro14 jeffboudier lunarflu alielfilali01 Weyaxi nickandbro danielhanchen mattmdjaga fffiloni silveriomf pcuenq carlosmoises

zaghadon marceloxp kramp Weyaxi terpjwu1 carlosmoises mathiasn1


@jinaai, we've recently launched an interesting model: https://huggingface.co/jinaai/jina-colbert-v1-en. In this post, I'd like to give you a quick introduction to ColBERT: the multi-vector search & late interaction retriever.

As you may already know, we've been developing embedding models such as https://huggingface.co/jinaai/jina-embeddings-v2-base-en for some time. These models, often called 'dense retrievers', generate a single representation for each document.

Embedding models like Jina-v2 have the advantage of quick integration with vector databases and good performance within a specific domain.

When discussing tasks within a specific domain, it means embedding models can perform very well by "seeing similar distributions". However, this also suggests that they might only perform "okay" on tasks outside of that domain and require fine-tuning.

Now, let's delve into multi-vector search and late-interaction models. The idea is quite simple:

1. During model training, you apply dimensionality reduction to decrease the vector dimensionality from 768 to 128 to save storage.
2. Now, with one query and one document, you match each query token embedding against every token embedding in the document to find the maximum similarity score. Repeat this process for each token in the query, from the second to the last, and then sum up all the maximum similarity scores.

This process is called multi-vector search because if your query has 5 tokens, you're keeping 5 * 128 token embeddings. The "max similarity sum-up" procedure is termed late interaction.

Multi-vector & Late interaction retrievers have the advantage of:

1. Excellent performance outside of a specific domain since they match at a token-level granularity.
2. Explainability: you can interpret your token-level matching and understand why the score is higher/lower.

Try our first multi-vector search at https://huggingface.co/jinaai/jina-colbert-v1-en and share your feedback!

clem osanseviero taylorbollman dev7halo victor dillfrescott nuggi sbrandeis


https://qwen.readthedocs.io/ 🔥 The official doc of Qwen1.5 is coming! This is a bilingual doc (English and Chinese, and it will be multilingual if I have time for them). The doc includes instructions for simple inference, running locally with GGUF, ollama, etc., quantization, finetuning, deployment, etc. We will continue adding more stuff to the doc. Stay tuned!


🚀🧙🏼♂Introducing OpenHermesPreferences: the largest open AI feedback dataset for RLHF & DPO

> Using LLMs to improve other LLMs, at scale!

Built in collaboration with the H4 Hugging Face team, it's a 1M preferences dataset on top of the amazing @teknium 's dataset.

Dataset:
https://huggingface.co/datasets/argilla/OpenHermesPreferences

The dataset is another example of open collaboration:

> The H4 team created responses with Mixtral using llm-swarm

> Argilla created responses with NousResearch Hermes-2-Yi-34B using distilabel

> The H4 ranked these responses + original response with PairRM from AllenAI, University of Southern California, Zhejiang University (@yuchenlin @DongfuTingle and colleagues)

We hope this dataset will help the community's research efforts towards understanding the role of AI feedback for LLM alignment.

We're particularly excited about the ability of filtering specific subsets to improve LLM skills like math or reasoning.

Here's how easy it is to filter by subset:

```
ds = load_dataset("HuggingFaceH4/OpenHermesPreferences", split="train")

# Get the categories of the source dataset
# ['airoboros2.2', 'CamelAI', 'caseus_custom', ...]
sources = ds.unique("source")

# Filter for a subset
ds_filtered = ds.filter(lambda x : x["source"] in ["metamath", "EvolInstruct_70k"], num_proc=6)
```

As usual, all the scripts to reproduce this work are available and open to the community!

https://huggingface.co/datasets/argilla/OpenHermesPreferences/tree/main

So fun collab between @vwxyzjn , @plaguss, @kashif, @philschmid &  @lewtun!

Open Source AI FTW!

davanstrien plaguss philschmid osanseviero alielfilali01 Rybens lewtun xiaotianhan gabrielmbmb clem vladbogo koutch kramp thivav Nindaleth

philschmid osanseviero alielfilali01 clem


MobileLLM: Optimizing Sub-billion Parameter Language Models for
  On-Device Use Cases (2402.14905)

MobileLLM

Optimizing Sub-billion Parameter Language Models for On-Device Use Cases

https://huggingface.co/papers/2402.14905

paper addresses the growing need for efficient large language models (LLMs) on mobile devices, driven by increasing cloud costs and latency concerns. We focus on designing top-quality LLMs with fewer than a billion parameters, a practical choice for mobile deployment. Contrary to prevailing belief emphasizing the pivotal role of data and parameter quantity in determining model quality, our investigation underscores the significance of model architecture for sub-billion scale LLMs. Leveraging deep and thin architectures, coupled with embedding sharing and grouped-query attention mechanisms, we establish a strong baseline network denoted as MobileLLM, which attains a remarkable 2.7%/4.3% accuracy boost over preceding 125M/350M state-of-the-art models. Additionally, we propose an immediate block-wise weight sharing approach with no increase in model size and only marginal latency overhead. The resultant models, denoted as MobileLLM-LS, demonstrate a further accuracy enhancement of 0.7%/0.8% than MobileLLM 125M/350M. Moreover, MobileLLM model family shows significant improvements compared to previous sub-billion models on chat benchmarks, and demonstrates close correctness to LLaMA-v2 7B in API calling tasks, highlighting the capability of small models for common on-device use cases.


Exciting news for `bitsandbytes`! We're thrilled to announce the release of the initial version of our new documentation: https://huggingface.co/docs/bitsandbytes/main/en/index .

Please let us know what you think: Your feedback is essential to us, and we would greatly appreciate any insights you have on how we can further enhance it or even better be happy to merge your contributions, filling in some blanks: Especially doc-strings are still a big topic and there several placeholder that would be super helpful to have filled in. Please post your feedback here: https://github.com/TimDettmers/bitsandbytes/discussions/1090

Since taking over maintenance together with Younes Belkada and since Hugging Face graciously agreed to support the library, we've already made enormous strides and community contributions have sprung back to life: It's so motivating to have so many knowledgeable contributors that often invest extensive free-time and bring their unique ideas to the table.

A notable example are our ongoing efforts to enable cross-platform support, including Intel, Apple Silicon, AMD, and Windows. Simultaneously, we're working diligently to streamline community contributions in BNB, making the process more accessible for everyone. A heartfelt thank you to all who have contributed thus far!

With HuggingFace's committed to supporting bitsandbytes going forward, we're sure to promptly respond to and integrate additional community contributions.

Looking forward to growing bitsandbytes further as part of the FOSS community: pushing forward the state of the art in democratization of AI!

Titus-von-Koeller clem osanseviero gsarti ybelkada mdouglas


First big community contribution on our evaluation suite, lighteval ⛅

@Ali-C137  added 3 evaluation tasks in Arabic:
- ACVA, a benchmark about Arabic culture
- MMLU, translated
- Exams, translated
(datasets provided/translated by the AceGPT team)

Congrats to them!
https://github.com/huggingface/lighteval/pull/44

clem alielfilali01 macadeliccc osanseviero vladbogo


"The principle of explainability of ai and its application in organizations"
Louis Vuarin, Véronique Steyer
—› 📔 https://doi.org/10.3917/res.240.0179

ABSTRACT: The explainability of Artificial Intelligence (AI) is cited in the literature as a pillar of AI ethics, yet few studies explore its organizational reality. This study proposes to remedy this shortcoming, based on interviews with actors in charge of designing and implementing AI in 17 organizations. Our results highlight: the massive substitution of explainability by the emphasis on performance indicators; the substitution of the requirement of understanding by a requirement of accountability; and the ambiguous place of industry experts within design processes, where they are employed to validate the apparent coherence of ‘black-box’ algorithms rather than to open and understand them. In organizational practice, explainability thus appears sufficiently undefined to reconcile contradictory injunctions. Comparing prescriptions in the literature and practices in the field, we discuss the risk of crystallizing these organizational issues via the standardization of management tools used as part of (or instead of) AI explainability.

Vuarin, Louis, et Véronique Steyer. « Le principe d’explicabilité de l’IA et son application dans les organisations », Réseaux, vol. 240, no. 4, 2023, pp. 179-210.

#ArtificialIntelligence #AIEthics #Explainability #Accountability

gsarti osanseviero Apparro Dlbk Alexandro14 clem Concor bukaka OmbelineM


Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity
  Tracking (2402.14811)

💥 Today in Interpretability & Analysis of LMs: Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking by @nikhil07prakash @tamarott @TalHaklay @belinkov @davidbau

Fine-tuning is commonly used to improve LM’s capabilities, but its impact on model-internal mechanisms remains poorly understood.

This work evaluates the impact of fine-tuning from a mechanistic perspective, using entity tracking in fine-tuned LLaMA 7B variants as a test bench.

Authors use path patching to highlight how fine-tuned models largely employ the same circuits as their pre-trained counterparts to solve entity tracking. Desiderata-based Component Masking (DCM) is used to discern the function of circuit components, finding that even the functionality of the circuit components remains consistent after fine-tuning.

Where do the gains stem from, then? Using Cross-Model Activation Patching (CMAP), authors show the benefits of fine-tuning are largely derived from an improved ability of circuit components to encode important task-relevant information rather than an overall functional rearrangement. Interestingly, fine-tuned activations are compatible with the base model despite no explicit constraint during representation learning.

🌐 Website: http://finetuning.baulab.info/
🤖 Model: https://huggingface.co/nikhil07prakash/float-7b
📄 Paper: https://huggingface.co/papers/2402.14811

🔍 All daily picks: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9


Genie

Generative Interactive Environments

https://huggingface.co/papers/2402.15391

introduce Genie, the first generative interactive environment trained in an unsupervised manner from unlabelled Internet videos. The model can be prompted to generate an endless variety of action-controllable virtual worlds described through text, synthetic images, photographs, and even sketches. At 11B parameters, Genie can be considered a foundation world model. It is comprised of a spatiotemporal video tokenizer, an autoregressive dynamics model, and a simple and scalable latent action model. Genie enables users to act in the generated environments on a frame-by-frame basis despite training without any ground-truth action labels or other domain-specific requirements typically found in the world model literature. Further the resulting learned latent action space facilitates training agents to imitate behaviors from unseen videos, opening the path for training generalist agents of the future.


A Closer Look at the Limitations of Instruction Tuning (2402.05119)

"A Closer Look at the Limitations of Instruction Tuning" is a new paper that explores the efficacy and limitations of Instruction Tuning (IT) in Large Language Models (LLMs) for conversational agents. The authors conduct a series of experiments using both LoRA fine-tuning (LFT) and standard full-parameter fine-tuning (SFT) across various LLMs and IT datasets.

The key findings are:
* LoRA fine-tuning (LFT) preserves the pre-training token distribution while SFT doesn't. This indicates that using LFT, post fine-tuning the model still heavily relies on the pre-training and doesn't acquire new information.
* Dataset scaling is ineffective for LFT - experiments show that scaling the dataset size 52x or even 326x doesn't improve the performance.
* LoRA fine-tuning mainly enhances response initiation and style without substantial knowledge enhancement.
* Full-parameter fine-tuning tends to degrade LLM knowledge base and increase hallucination occurrences.
* Popular other methods and adjustments fail to significantly outperform simple LoRA fine-tuned models in terms of conversational quality and accuracy.

Congrats to the authors @Sreyan88 and others for their work!

Paper: https://huggingface.co/papers/2402.05119

Hopkinx osanseviero ioanacroi kaleinaNyan clem Felladrin RamaniD sonalkum TheDetective Sreyan88 Arjun4707 TerraNull schoolta


🌟✨ Exciting Announcement: NVIDIA AI Foundation Models ✨🌟

🚀 Interact effortlessly with the latest SOTA AI model APIs, all optimized on the powerful NVIDIA accelerated computing stack-right from your browser! 💻⚡

🔗 Web Page: https://catalog.ngc.nvidia.com/ai-foundation-models

🌟🎯 Favorites:

🔹 Code Generation:
1⃣ Code Llama 70B 📝🔥: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/codellama-70b
Model 🤖: https://huggingface.co/codellama/CodeLlama-70b-hf

🔹 Text and Code Generation:
1⃣ Gemma 7B 💬💻: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/gemma-7b
Model 🤖: https://huggingface.co/google/gemma-7b
2⃣ Yi-34B 📚💡: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/yi-34b
Model 🤖: https://huggingface.co/01-ai/Yi-34B

🔹 Text Generation:
1⃣ Mamba-Chat 💬🐍: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/mamba-chat
Model 🤖: https://huggingface.co/havenhq/mamba-chat
2⃣ Llama 2 70B 📝🦙: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/llama2-70b
Model 🤖: https://huggingface.co/meta-llama/Llama-2-70b

🔹 Text-To-Text Translation:
1⃣ SeamlessM4T V2 🌐🔄: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/seamless-m4t2-t2tt
Model 🤖: https://huggingface.co/facebook/seamless-m4t-v2-large

🔹 Image Generation:
1⃣ Stable Diffusion XL 🎨🔍: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/sdxl

🔹 Image Conversation:
1⃣ NeVA-22B 🗨📸: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/neva-22b

🔹 Image Classification and Object Detection:
1⃣ CLIP 🖼🔍: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/clip

🔹 Voice Conversion:
1⃣ Maxine Voice Font 🗣🎶: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/voice-font

🔹 Multimodal LLM (MLLM):
1⃣ Kosmos-2 🌐👁: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/kosmos-2

MiSTe-R vladbogo ssbagpcm osanseviero samusenps Josemalyson eramax s3nh Alexandro14 Csplk ArunkumarVR xiaotianhan Samoed shubhamg2208 Antreas dillfrescott yahma alielfilali01 Nymbo


LLM Agents can Autonomously Hack Websites (2402.06664)

"LLM Agents can Autonomously Hack Websites" is a new paper that investigates the capacity of LLMs to autonomously execute cybersecurity attacks on websites, such as SQL injections without human guidance.

Key points:
* It uses a LLM integrated with Playwright, a headless web browser, enabling automated web interactions through function calling.
* It gives access to the LLM to 7 web hacking documents and planning capabilities through specific prompting, without disclosing the exact methods to prevent misuse.

GPT-4 achieves a 73.3% success rate on the tested vulnerabilities, emphasizing the potential cybersecurity risks posed by advanced LLMs. Other open models cannot yet perform these types of attacks (results in screenshot).

Congrats to the authors for their work!

Paper: https://huggingface.co/papers/2402.06664

Kukedlc alwint3r osanseviero leeloolee theainerd WhenceFade


VLMs have a resolution problem, which prevents them from finding small details in large images. In my community blog post, I discuss the ways to solve it and describe the details of MC-LLaVA architecture - https://huggingface.co/blog/visheratin/vlm-resolution-curse

Check it out, and let me know what you think!

osanseviero samusenps georgechkhvirkia clem radames duxx Alexandro14 Molbap oldsea


Today, I’m thrilled to release a project I’ve been working on for the past couple weeks in collaboration with Hugging Face: the TTS Arena.

The TTS Arena, inspired by LMSys's Chatbot Arena, allows you to enter text which will be synthesized by two SOTA models. You can then vote on which model generated a better sample. The results will be published on a publicly-accessible leaderboard.

We’ve added several open access models, including Pheme, MetaVoice, XTTS, OpenVoice, & WhisperSpeech. It also includes the proprietary ElevenLabs model.

If you have any questions, suggestions, or feedback, please don’t hesitate to DM me on X (https://twitter.com/realmrfakename) or open a discussion in the Space. More details coming soon!

Try it out: https://huggingface.co/spaces/TTS-AGI/TTS-Arena

tomaarsen Korakoe mlabonne samusenps clem kurianbenoy reach-vb Alexandro14 Wauplin realsammyt mvaloatto julien-c ameerazam08 clefourrier alielfilali01 Suparious michaljunczyk Nymbo

rafaelpierrehf vladbogo Korakoe clem reach-vb mathiasn1 ameerazam08 Nymbo


Check out the demo of SPIN-Diffusion made by @angelahzyuan at: https://huggingface.co/spaces/UCLA-AGI/SPIN-Diffusion-demo-v1

osanseviero angelahzyuan MaziyarPanahi samusenps akhaliq mohammed-17


VideoPrism: A Foundational Visual Encoder for Video Understanding (2402.13217)

VideoPrism is a new video encoder that improves video understanding through a unique training strategy, using a vast dataset (36 million high-quality video-caption pairs and 582 million video clips) for comprehensive learning.

Key points:
* It employs a two-stage training approach, initially aligning video and text encoders, followed by an enhanced video-only masked autoencoding process to learn appearance and motion.
* It achieves superior performance in a wide array of tasks, such as general video understanding, zero-shot video-text retrieval, video captioning, QA, and computer vision for science, having top performance on 30 out of 33 benchmarks.

Congrats to the authors for their work!

Paper: https://huggingface.co/papers/2402.13217

Kukedlc guitarliu0320 osanseviero clem liusuchao Alexandro14 clefourrier


OpenCodeInterpreter: Integrating Code Generation with Execution and
  Refinement (2402.14658)

OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement

https://huggingface.co/papers/2402.14658

The introduction of large language models has significantly advanced code generation. However, open-source models often lack the execution capabilities and iterative refinement of advanced systems like the GPT-4 Code Interpreter. To address this, we introduce OpenCodeInterpreter, a family of open-source code systems designed for generating, executing, and iteratively refining code. Supported by Code-Feedback, a dataset featuring 68K multi-turn interactions, OpenCodeInterpreter integrates execution and human feedback for dynamic code refinement. Our comprehensive evaluation of OpenCodeInterpreter across key benchmarks such as HumanEval, MBPP, and their enhanced versions from EvalPlus reveals its exceptional performance. Notably, OpenCodeInterpreter-33B achieves an accuracy of 83.2 (76.4) on the average (and plus versions) of HumanEval and MBPP, closely rivaling GPT-4's 84.2 (76.2) and further elevates to 91.6 (84.6) with synthesized human feedback from GPT-4. OpenCodeInterpreter brings the gap between open-source code generation models and proprietary systems like GPT-4 Code Interpreter.

Jenish-23 mrfakename radames Shawnlight Kukedlc vladbogo DmitryRyumin boapps nirajandhakal


🤗 Sentence Transformers v2.4.0 for embedding models is now out! It introduces a lot of powerful features, such as:

1. Matryoshka Loss function - you can now train & perform inference on 🪆 Matryoshka Embedding models. See also our blogpost: https://huggingface.co/blog/matryoshka

2. CoSENTLoss & AnglELoss: State of the art loss functions. These are quite interesting, they outperform CosineSimilarityLoss on nearly all benchmarks as a drop-in replacement! See also the docs: https://sbert.net/docs/package_reference/losses.html#cosentloss

3. Prompt templates: Many popular models such as https://huggingface.co/intfloat/multilingual-e5-large and https://huggingface.co/BAAI/bge-large-en-v1.5 prefix their texts with prompts, so this adds configuration options to automatically include prompts using `model.encode(..., prompt_name="query")` which will include a prompt with the name "query". More info in the docs: https://sbert.net/examples/applications/computing-embeddings/README.html#prompt-templates

4. Instructor support: Support for the INSTRUCTOR line of models, such as https://huggingface.co/hkunlp/instructor-large. Learn how to use them here: https://sbert.net/docs/pretrained_models.html#instructor-models

5. Removed NLTK & sentencepiece dependencies: Should allow for a smaller installation & a slightly faster import!

6. Updated documentation: a new Loss Overview section: https://sbert.net/docs/training/loss_overview.html and more detailed loss functions: https://sbert.net/docs/package_reference/losses.html

And much more! See the full release notes here: https://github.com/UKPLab/sentence-transformers/releases/tag/v2.4.0

Some more very exciting updates are still on the horizon!

radames Azamat1k Kukedlc eramax victor alvarobartt asoria bfn DeepMount00


Just when I was about to go to bed....... Here we go again

osanseviero lysandre reach-vb mathiasn1 pcuenq jpm8998 riddhiman victor julien-c radames vladbogo Tom-Neverwinter Azamat1k rafaelpierrehf himasai9711 burtugeey sbrandeis

regisss ybelkada osanseviero reach-vb riddhiman clem julien-c radames


Compared Stable Diffusion 3 with Dall-E3 and results are mind blowing.

SD 3 can follow prompts many times better than SD 1.5 or SDXL. It is even better than Dall-E3 in following text / spelling prompts.

The realism of the SD 3 can't be even compared with Dall-E3, since every Dall-E3 output is like a digital render.

Can't wait to get approved of Stability AI early preview program to do more intensive testing.

Some people says be skeptical about cherry picking. I agree but I hope that these Stability AI released images are not that heavy cherry picking.

You can see SD3 vs Dall-E3 comparison here : https://youtu.be/DJxodszsERo

Dlbk mdzihad vladbogo Kukedlc admarcosai danielus Rsln SANCHEZRICK KADS-dev Glavin001 NHLOCAL


YOLOv9: Learning What You Want to Learn Using Programmable Gradient
  Information (2402.13616)

🎉✨ Exciting Research Alert! YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information 🚀

YOLOv9 is the latest breakthrough in object detection!

📄 Title: YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information

👥 Authors: Chien-Yao Wang et al.
📅 Published: ArXiv, February 2024

🔗 Paper: https://huggingface.co/papers/2402.13616
🔗 Model 🤖: https://huggingface.co/adonaivera/yolov9
🔗 Repo: https://github.com/WongKinYiu/yolov9

🚀 Don't miss out on this cutting-edge research! Explore YOLOv9 today and stay ahead of the curve in the dynamic world of computer vision. 🌟

🔍 Keywords: #YOLOv9 #ObjectDetection #DeepLearning #ComputerVision #Innovation #Research #ArtificialIntelligence

Kukedlc samusenps osanseviero taufiqdp Dlbk mathiasn1 kadirnar vladbogo dillfrescott jairo

samusenps osanseviero clem Dlbk kadirnar dillfrescott femboysLover ntdas


Hugging Face announces Cosmo 1B, a fully open sourced Phi competitor with an open sourced dataset. The dataset references various articles and textbooks as "seed data" to generate conversations. Licensed under the Apache 2.0 license. The dataset, dubbed "Cosmopedia," is published on the Hugging Face Hub under the Apache 2.0 license. It was generated using Mixtral 8x7B with various sources (AutoMathText, OpenStax, WikiHow, etc) as "seed data."

Model: https://huggingface.co/HuggingFaceTB/cosmo-1b
Dataset: https://huggingface.co/datasets/HuggingFaceTB/cosmopedia

Kukedlc wesley-mtk osanseviero clem kramp Dlbk radames admarcosai compressionsavant Nymbo

samusenps osanseviero clem Dlbk s3nh radames MaziyarPanahi


Enhanced Hallucination Detection in Neural Machine Translation through
  Simple Detector Aggregation (2402.13331)

🔍 Today's pick in Interpretability & Analysis of LMs: Enhanced Hallucination Detection in Neural Machine Translation through Simple Detector Aggregation by A. Himmi G. Staerman M. Picot @Colombo @nunonmg

Previous work on hallucination detection for MT showed that different detectors excel at detecting different types of hallucinations.

In this context, detectors based solely on model internals such as input contributions or sequence log-probabilities fare well on fully-detached hallucinations but show limited performances on oscillatory hallucinations, where ad-hoc trained detectors are still the best-performing methods.

This work proposes Simple Detectors Aggregation (STARE), an aggregation procedure to leverage detectors’ complementary strengths. Authors experiment with two popular hallucination detection benchmarks (LFAN-HALL and HalOmi), showing that STARE outperforms single detectors and other aggregation baselines.

Results obtained aggregating internal detectors highlight how model-based features that are readily available as generation byproducts can outperform computationally expensive ad-hoc solutions.

📄 Paper: https://huggingface.co/papers/2402.13331

🔍 All daily picks in LM interpretability: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9

samusenps lunarflu osanseviero Kukedlc nunonmg Alexandro14


LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (2402.13753)

LongRoPE

Extending LLM Context Window Beyond 2 Million Tokens

https://huggingface.co/papers/2402.13753

Large context window is a desirable feature in large language models (LLMs). However, due to high fine-tuning costs, scarcity of long texts, and catastrophic values introduced by new token positions, current extended context windows are limited to around 128k tokens. This paper introduces LongRoPE that, for the first time, extends the context window of pre-trained LLMs to an impressive 2048k tokens, with up to only 1k fine-tuning steps at within 256k training lengths, while maintaining performance at the original short context window. This is achieved by three key innovations: (i) we identify and exploit two forms of non-uniformities in positional interpolation through an efficient search, providing a better initialization for fine-tuning and enabling an 8x extension in non-fine-tuning scenarios; (ii) we introduce a progressive extension strategy that first fine-tunes a 256k length LLM and then conducts a second positional interpolation on the fine-tuned extended LLM to achieve a 2048k context window; (iii) we readjust LongRoPE on 8k length to recover the short context window performance. Extensive experiments on LLaMA2 and Mistral across various tasks demonstrate the effectiveness of our method. Models extended via LongRoPE retain the original architecture with minor modifications to the positional embedding, and can reuse most pre-existing optimizations.

eramax Madisen vladbogo PrathmeshZ clem NickyNicky lunarflu osanseviero thughost Ji-Ha admarcosai PedramR xiaotianhan

tomaarsen FranZuzz omaryshchenko clem lunarflu osanseviero Ji-Ha charlieCs


Data map of the languages of https://huggingface.co/datasets/CohereForAI/aya_dataset

osanseviero clem TuringsSolutions samusenps lunarflu kurianbenoy sayhan not-lain lilouuch AliArshad shivi OmbelineM


Rephrasing the Web: A Recipe for Compute and Data-Efficient Language
  Modeling (2401.16380)

Web Rephrase Augmented Pre-training (WRAP) enhances language model training efficiency by transforming documents into structured formats.

Key aspects:
* Utilizes an instruction-tuned model to rephrase web content into styles such as Wikipedia or Q/A, creating a blend of synthetic and real data for training.
* Demonstrated improvements of over 10% better perplexity, alongside more than 2% increase in zero-shot question-answering accuracy.

Congrats to the authors for their work!

Paper: https://huggingface.co/papers/2401.16380

osanseviero clem lunarflu BroCodeR Alexandro14 kotyKD abdullah


📚🔎 If you're building RAG applications, you should check this out:

⚙ I've built a new space to let you visualize the chunks you get with different text splitting methods!

➡ Visualize your chunks here:
https://huggingface.co/spaces/m-ric/chunk_visualizer

MoritzLaurer vladbogo DmitryRyumin SolarWindRider andrewrreed aari1995 osanseviero clem lunarflu sds20 sparkonelabs mushimeraki


looking at the tokenizer and the naming (“_en“), Google Gemma is very likely to have a multilingual counterpart. 👀

Thoughts?

vladbogo sayhan osanseviero clem noobmldude


I am thrilled to announce Gemma, new 2B and 7B models from Google, based on the same research and technology used to train the Gemini models! These models achieve state-of-the-art performance for their size, and are launched across Transformers, Google Cloud, and many other surfaces worldwide starting today.

Get started using and adapting Gemma in the model Collection: https://huggingface.co/collections/google/gemma-release-65d5efbccdbb8c4202ec078b

These launches are the product of an outstanding collaboration between the Google DeepMind and Hugging Face teams over the last few months -- very proud of the work both teams have done, from integration with Vertex AI to optimization across the stack. Read more about the partnership in the main launch by @philschmid @osanseviero @pcuenq on the launch blog: https://huggingface.co/blog/gemma

More information below if you are curious about training details, eval results, and safety characteristics!

Gemma Tech Report: https://goo.gle/GemmaReport
Launch announcement: https://blog.google/technology/developers/gemma-open-models/

algoartist megrisdal andysalerno osanseviero victor BrigitteTousi alvarobartt abidlabs clem jeffboudier joaogante DmitryRyumin manvinder kramp VictorSanh Alexandro14 samforeman vladbogo taufiqdp pcuenq Kukedlc Madisen chansung MoritzLaurer m-ric andrewrreed Anthonyg5005 abhishek mathiasn1 nicolasdec Cstark ethix

alvarobartt abidlabs clem thibaultM osanseviero AP123 jeffboudier joaogante VictorSanh Alexandro14 pcuenq Kukedlc Madisen Bils chansung SanRosenberg andrewrreed Bolamide ZenithBrew allknowingroger abhishek sugatoray

clem osanseviero VictorSanh Kukedlc Madisen chansung abhishek

osanseviero Kukedlc Madisen abhishek Cstark


LLM hallucination detection papers be like:

* check the image to get the joke 👀

fahnub Kukedlc Doma02 gsarti sayhan MarinaraSpaghetti vijay8642 dashfunnydashdash Vermilion3154 Kirkalish notoookay


 Neural Network Diffusion

https://huggingface.co/papers/2402.13144

Diffusion models have achieved remarkable success in image and video generation. In this work, we demonstrate that diffusion models can also generate high-performing neural network parameters. Our approach is simple, utilizing an autoencoder and a standard latent diffusion model. The autoencoder extracts latent representations of a subset of the trained network parameters. A diffusion model is then trained to synthesize these latent parameter representations from random noise. It then generates new representations that are passed through the autoencoder's decoder, whose outputs are ready to use as new subsets of network parameters. Across various architectures and datasets, our diffusion process consistently generates models of comparable or improved performance over trained networks, with minimal additional cost. Notably, we empirically find that the generated models perform differently with the trained networks. Our results encourage more exploration on the versatile use of diffusion models.

osanseviero kgourgou SanRosenberg vladbogo gsarti Salwa-Zeitoun SuperShark takeraparterer DataSoul


New base pretrained models on the Open LLM Leaderboard!

Two new OSS models by Google, who's getting back in the game 😎
The 7B is 2nd of the leaderboard, and better than Mistral (notably on GSM8K, aka math).

https://huggingface.co/google/gemma-7b
https://huggingface.co/google/gemma-2b

Check more results on the leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard

tomaarsen Citaman Sylvestre beomi eramax manvinder macaulj1 osanseviero lazarustda Alexandro14 kramp Rayabarapu clem trisfromgoogle VictorSanh vladbogo mrfakename Kukedlc johndpope alielfilali01 BrigitteTousi Rybens


Exciting news for Indic LLMs! 🚀

Sarvam AI just released high-quality, curated dataset with multi-turn conversations in English, Hindi, and Hinglish! 💎 With a whopping 100K samples! 🤯
Check it out: https://huggingface.co/datasets/sarvamai/samvaad-hi-v1

Who's going to finetune high-quality SFT models on this dataset? ✨
if you are interested in pushing the boundaries with respect to Indic LLMs, join the discord channel: https://discord.gg/hugging-face-879548962464493619

osanseviero rajveer43 Rayabarapu clem masterfury alielfilali01


Backward Lens: Projecting Language Model Gradients into the Vocabulary
  Space (2402.12865)

🔍 Today's pick in Interpretability & Analysis of LMs: Backward Lens: Projecting Language Model Gradients into the Vocabulary Space by @shaharkatz @belinkov @mega @liorwolf

Recent interpretability works explore intermediate model representations by projecting them to vocabulary space. This work explores projecting gradients computed from the backward pass to vocabulary space to explain how a single forward-backward pass edits LM knowledge.

Authors identify a mechanism they dub “imprint and shift” in the forward module in transformer layer. Specifically, the “imprint” refers to the first layer, to or from which the learning process adds or subtracts copies of the intermediate inputs encountered during the forward pass. The “shift” refers to the second matrix, where the weights are shifted by the embedding of the target token.

Authors note that the dominant components in constructing gradients are derived from the outer product of the last token’s input and the Vector-Jacobian Product, and that the latter contains the embedding of the target token.

In light of this, a new editing approach named “forward pass shifting” is proposed to update the shifting component of a layer’s feedforward module without backpropagation, using only layer inputs and target token embeddings. The method performs on par with significantly more expensive editing approaches like ROME for single-fact editing, but is less robust to paraphrasing.

Authors note that these results provide promising evidence on the possibility of finding shortcuts to fine-tuning by directly injecting knowledge in model layers.

📄 Paper: https://huggingface.co/papers/2402.12865

🔍 All daily picks in LM interpretability: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9


FasterViT: Fast Vision Transformers with Hierarchical Attention (2306.06189)

🚀🔥🌟 New Research Alert - ICLR 2024! 🌟🔥🚀
📄 Title: FasterViT: Fast Vision Transformers with Hierarchical Attention

👥 Authors: @ahatamiz, @slivorezzz et al.

📅 Conference: ICLR, May 7-11, 2024 | Vienna, Austria 🇦🇹

🔗 Paper: https://huggingface.co/papers/2306.06189

🔗 Model 🤖 : https://huggingface.co/nvidia/FasterViT
🔗 Repo: https://github.com/NVlabs/FasterViT

📚 More Papers: more cutting-edge research presented at other conferences in the https://huggingface.co/spaces/DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin

🔍 Keywords: #VisionTransformers #DeepLearning #ComputerVision #ICLR2024 #MachineLearning #HierarchicalAttention #NeuralNetworks #Research #ArtificialIntelligence #Innovation

osanseviero merve Alexandro14 clem ahatamiz chansung Falah dillfrescott


REALIGN is a new method designed to improve the alignment of Large Language Models (LLMs) with human values by reformatting instruction data. This approach enhances LLM performance across various metrics by aligning responses with predefined criteria and evidence.

Key points:

* REALIGN has three steps: criteria definition, retrieval augmentation, and response reformatting
* It rewrites pairs (query, response) to enhance data quality for fine-tuning LLMs.
* It has shown significant improvements in general alignment, math reasoning and other tasks.

Congrats to the authors for their work!

Paper: https://huggingface.co/papers/2402.12219
Code: https://github.com/GAIR-NLP/ReAlign


MANIFESTO
After working in fashion e-commerce for years I've come to the conclusion that in e-commerce we do not sell clothes... we sell images of clothes. Compressed, digital versions of physical products. As Roland Barthes pointed out in The Fashion System, a product image is a symbol or metaphor of a product. Media--in this case images--mediates the space between customer and product; viewer and object. Images can be altered, changed, corrupted, photoshopped, edited, deleted, or imagined. E-commerce products (or e-commerce photos) can thought of as a possibility space of digital pixels. AI/ML can analyze, manipulate, and create within this "possibility space of pixels"--thus it can be observed that there are opportunities to intervene in the physical fashion world through the imagination of artificial intelligence. Not to replace human creativity--but to augment it. To make it ART-ificial. Art is an artificial representation of reality. AI images are an artificial representation of reality. The sewing  machine greatly increased the efficiency of clothing production. Similarly, AI has greatly increased efficiency image production, in our case product photo production. The fashion design paradigm of the past century (design->produce->photograph) has been flipped on this head. Instead of going from physical clothing to digital image via photography--we can go from digital image to physical clothing via stable diffusion. We are writing the chapter of Understanding Media that Marshall McLuhan never imagined. Virtual production hasn't replaced the physical production; it has simply made it out of style.

fffiloni TuringsSolutions vladbogo dcsdcsdasdasa osanseviero Taf2023 aidystark kramp samusenps angelium111 Madisen Wauplin lunarflu pascalnjue Chieh-Yun sghosh2 DocRace ozgung Krishana671 Jia-ao Nymbo strangebose2 marinarosa MustardWhale516

varadpuntambekar strangebose2 Nymbo marinarosa

Максим Николаевич Подъяпольский


Fine-grained Hallucination Detection and Editing for Language Models (2401.06855)

Fantastic Beasts (*Hallucinations*) and Where to Find Them 🔎🧌

This paper breaks down LLM hallucinations into six different types:

1⃣ Entity: Involves errors in nouns. Changing that single entity can make the sentence correct.

2⃣ Relation: Involves errors in verbs, prepositions, or adjectives. They can be fixed by correcting the relation.

3⃣ Contradictory: Sentences that contradict factually correct information.

4⃣ Invented: When the LLM generates sentences with concepts that don't exist in the real world.

5⃣ Subjective: When the LLM generates sentences influenced by personal beliefs, feelings, biases, etc.

6⃣ Unverifiable: When the LLM comes up with sentences containing information that can't be verified. E.g., Personal or private matters.

The first two types of hallucinations are relatively easy to correct, given that we can rewrite them by changing the entity or relation. However, the other four would mostly need to be removed to make the sentence factually correct.

Paper:  https://huggingface.co/papers/2401.06855

Kukedlc vladbogo bisnotforbella osanseviero jeffboudier andrewrreed EpsilonPhoenix7 loudog2323 Hanyu66


ICYMI! Nomic Embed v1.5: Resizable Production Embeddings with Matryoshka Representation Learning

- Variable embedding dimension from 64 <-> 768
- Outperforms text-embedding-ada-002 while achieving a 3x memory reduction
- Day 1 integrations with Langchain, LlamaIndex, MongoDB, and Sentence Transformers

Check out
https://huggingface.co/nomic-ai/nomic-embed-text-v1.5 for the model weights.

Technical report: https://static.nomic.ai/reports/2024_Nomic_Embed_Text_Technical_Report.pdf
Blog Post: https://blog.nomic.ai/posts/nomic-embed-matryoshka
Original Tweet Thread: https://x.com/nomic_ai/status/1757782157374734665?s=20

philschmid Xenova osanseviero Taf2023 davanstrien sayhan linhphanff Yhyu13


Working through the Reddit dataset, one thing that occurs to me is we pretty much always train LLMs to be a conversation between 2 parties like Bot/Human or Instruction/Response.

It seems far more common with internet data that we have multi-speaker/group discussions with a dynamic number of speakers. This also seems to be more realistic to the real world too and requires a bit more understanding to model.

Is there some research into this? I have some ideas of how I'd like to implement it, but I wonder if some work has already been done here?

yungwarlock Moooritz krutarth53 pranav-deshpande ClaudioItaly Stephenmcgurrin sthakur2 poetofgrindset Noomam Luxter77


Speculative Streaming: Fast LLM Inference without Auxiliary Models (2402.11131)

Speculative Streaming

Fast LLM Inference without Auxiliary Models

https://huggingface.co/papers/2402.11131

Speculative decoding is a prominent technique to speed up the inference of a large target language model based on predictions of an auxiliary draft model. While effective, in application-specific settings, it often involves fine-tuning both draft and target models to achieve high acceptance rates. As the number of downstream tasks grows, these draft models add significant complexity to inference systems. We propose Speculative Streaming, a single-model speculative decoding method that fuses drafting into the target model by changing the fine-tuning objective from next token prediction to future n-gram prediction. Speculative Streaming speeds up decoding by 1.8 - 3.1X in a diverse set of tasks, such as Summarization, Structured Queries, and Meaning Representation, without sacrificing generation quality. Additionally, Speculative Streaming is parameter-efficient. It achieves on-par/higher speed-ups than Medusa-style architectures while using ~10000X fewer extra parameters, making it well-suited for resource-constrained devices.


I'm working on a templated Quarto website and was looking for some good placeholder content. Do you have any ideas? The ideal content would be:
- Widely legible
- Short
- Include long form text
- Would benefit from some interactive applications


We fine-tuned 25 mistral-7b models that outperform GPT-4 on task-specific use cases!

Check them out at LoRA Land: https://pbase.ai/3uFh7Qc

You can prompt them and compare their results to mistral-7b-instruct in real time!

They're also on HF for you to play with: https://huggingface.co/predibase

Let us know what you think!

Kukedlc clem arnavgrg samusenps gsarti cnmoro anthonymikinka chandlerTSLabs lunarflu Dlbk dillfrescott mathiasn1 steventrouble

9bow kramp Pent chandlerTSLabs dillfrescott


🔥 New LLM leaderboard blog: Open Ko LLM!

One of the oldest leaderboards on the hub, it has already evaluated more than 1000 models! It uses Korean translations of MMLU, ARC, HellaSwag, TruthfulQA, and a new dataset, Korean CommonGen, about specific common sense alignement.

https://huggingface.co/spaces/upstage/open-ko-llm-leaderboard

What's interesting about this leaderboard is how it drove LLM development in Korea, with on average about 4 submissions/models per day since it started!
Really looking forward to seeing similar initiatives in other languages, to help qualitative models emerge outside of "just English" (for the other 2/3rds of the world).

Read more about how the leaderboard in the intro blog: https://huggingface.co/blog/leaderboards-on-the-hub-upstage
Congrats to @Chanjun , @hunkim and the Upstage team!

Chanjun kramp KnutJaegersberg clem vladbogo jeffboudier sbrandeis


In-Context Learning Demonstration Selection via Influence Analysis (2402.11750)

🔍 Today's pick in Interpretability & Analysis of LMs: In-Context Learning Demonstration Selection via Influence Analysis
 by Vinay M.S. @minhhaovan X. Wu

Recent work showed how the performance of LMs using in-context learning (ICL) is heavily dependent on selected demonstrations.

This work introduces InfICL, a demonstration selection method using influence functions to identify salient training examples to use as demonstrations at inference time. InfICL is tested alongside other examples selection baselines for prompting medium-sized LLMs for COLA and RTE, showing improvements over other methods especially when a smaller number of in-context examples is used.

📄 Paper: https://huggingface.co/papers/2402.11750

🔍 All daily picks in LM interpretability: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9

kramp clem vladbogo santiviquez nickandbro


📢 New Research Alert - AAAI 2024!
📄 Title: CARAT: Contrastive Feature Reconstruction and Aggregation for Multi-Modal Multi-Label Emotion Recognition

👥 Authors: Cheng Peng et al.

📅 Conference: AAAI, February 20-27, 2024 | Vancouver, Canada 🇨🇦

🔗 Paper: https://arxiv.org/abs/2312.10201

🔗 Repository: https://github.com/chengzju/CARAT

📚 More Papers: Explore a collection of exciting papers presented at AAAI 2024 and other conferences in the repositories:
- AAAI 2024 Papers: https://github.com/DmitryRyumin/AAAI-2024-Papers, @DmitryRyumin
- Other Conferences: https://huggingface.co/spaces/DmitryRyumin/NewEraAI-Papers

🔍 Keywords: #EmotionRecognition #MultiModal #AI #Research #AAAI2024 #MachineLearning


Isn't it sad that VLMs don't have any inference parameters for the vision part? Well, MC-LLaVA now has two whole knobs you can use to make it find even the smallest details! I finally (almost) properly implemented multi-crop, and now you can control the number of crops and how many image tokens will be generated. The video shows how, by increasing the number of crops and tokens, my 3B model correctly identifies the 30x90 pixel logo in the 3200x3000 pixel image.
Other notable updates:
- I use SigLIP from Transformers, so you don't need to install additional libraries.
- the model now supports auto classes, so you can create the model and processor with only two lines.
- performance increased by 10%+ across all benchmarks.

The work is far from over, but it feels like good progress.

The model on the hub: https://huggingface.co/visheratin/MC-LLaVA-3b
You can try the model here: https://huggingface.co/spaces/visheratin/mc-llava-3b

samusenps osanseviero Sylvestre lusstta kinetical dillfrescott Azamat1k Alexandro14

samusenps clem lusstta okbottom dillfrescott


Hi, welcome to my first post here!

I am slowly wrangling about 5 years of reddit comments (2015-2020). It's a total of billions samples that can be filtered as comment-reply pairs, chains of discussion, filtered by subreddit, up/down votes, controversy, sentiment, and more.

Any requests or ideas for curated datasets from here? I'll also tinker with uploading the entire dataset potentially in chunks or something, but it's quite a few terabytes in total, so I'll need to break it up still. I have some ideas for datasets I personally want too, but curious if anyone has something they'd really like to see that sounds interesting too.

osanseviero Dewa clem VictorSanh kramp mrfakename samusenps Xenova qnguyen3 gsarti alielfilali01 MarinaraSpaghetti Noomam

osanseviero Dewa clem isavita samusenps ivanfioravanti


🚀 Exciting news from 🤗 PEFT!

We are introducing new merging methods for LoRA adapters. These methods allow for retaining the unique capabilities of individual LoRAs while enabling them to combine their strengths: https://huggingface.co/blog/peft_merging

We explored the application of merging LoRA adapters in the context of personal code copilot before 🚀👾✨. Please go through the below thread on it:  https://x.com/sourab_m/status/1718008115726283004?s=20

New merging methods `ties`, `dare`, and `magnitude_prune` introduced alongside existing methods `cat`, `linear`, and `svd`. Blogpost details each method. These methods can be applied on-the-fly during inference time instead of merging offline enabling great developer UX. ✨

How do I merge my LoRA adapters?
Easy, use class method `add_weighted_adapter()`. For example, below you can see how we can combine three LoRA adapters using `ties` method. We can observe that merged adapter can retain the capabilities of individual adapters!

Now that we have seen they can retain individual LoRAs, how about use cases wherein we require the capabilities from multiple LoRAs being merged/combined?  Below is an application of it in text-to-image domain. 🖼

Kudos to @prateeky2806 (TIES author) and Le Yu (DARE author) for their kind and generous guidance on the PRs! Also, if you want to explore full model merging, refer to super cool projects like https://github.com/arcee-ai/mergekit/tree/main,  https://github.com/Gryphe/BlockMerge_Gradient and https://github.com/yule-BUAA/MergeLM/tree/main.

Excited to see what the community creates on top of this! 🚀✨ #LetsBuildTogether

jeffboudier samusenps clem ybelkada Kukedlc boapps vWake SnehShah impactframes Citaman MexIvanov ClayFace vanillaOVO madstuntman11 bachvudinh

kramp lysandre osanseviero vladbogo samusenps clem ybelkada impactframes madstuntman11


In Search of Needles in a 10M Haystack

Recurrent Memory Finds What LLMs Miss

paper addresses the challenge of processing long documents using generative transformer models. To evaluate different approaches, we introduce BABILong, a new benchmark designed to assess model capabilities in extracting and processing distributed facts within extensive texts. Our evaluation, which includes benchmarks for GPT-4 and RAG, reveals that common methods are effective only for sequences up to 10^4 elements. In contrast, fine-tuning GPT-2 with recurrent memory augmentations enables it to handle tasks involving up to 10^7 elements. This achievement marks a substantial leap, as it is by far the longest input processed by any open neural network model to date, demonstrating a significant improvement in the processing capabilities for long sequences


Recovering the Pre-Fine-Tuning Weights of Generative Models (2402.10208)

Spectral DeTuning is a new method that successfully recovers the original weights of generative models before they were fine-tuned with human feedback or other customization. It shows that pre-fine-tuning weights are recoverable and methods fine-tuned with LoRA can be susceptible to a new weight recovery type of attack.

Key aspects of the paper:
	•	It introduces Spectral DeTuning for reversing Low-rank Adaptation (LoRA) fine-tuning, targeting original weight restoration through spectral analysis.
	•	LoWRA Bench Dataset: It introduces a dataset for testing Spectral DeTuning across various models and tasks, featuring extensive layers for comprehensive evaluation.
	•	It reveals LoRA fine-tuned models' vulnerability to weight recovery attacks, questioning the security of fine-tuning modifications.

Congrats to the authors for their work!

Paper: https://huggingface.co/papers/2402.10208
Dataset: https://huggingface.co/datasets/Eliahu/LoWRA-Bench
Project page: https://vision.huji.ac.il/spectral_detuning/
Code: https://github.com/eliahuhorwitz/Spectral-DeTuning

osanseviero PeepDaSlan9 samusenps clem deepkyu Svngoku sayhan ababio


YOLO-World: Real-Time, Zero-Shot Object Detection 🔥 🔥 🔥

YOLO-World was designed to solve a limitation of existing zero-shot object detection models: speed. Whereas other state-of-the-art models use Transformers, a powerful but typically slower architecture, YOLO-World uses the faster CNN-based YOLO architecture.

YOLO-World provides three models: small with 13M (re-parametrized 77M), medium with 29M (re-parametrized 92M), and large with 48M (re-parametrized 110M) parameters.

The YOLO-World team benchmarked the model on the LVIS dataset and measured their performance on the V100 without any performance acceleration mechanisms like quantization or TensorRT.

According to the paper, YOLO-World reached 35.4 AP with 52.0 FPS for the L version and 26.2 AP with 74.1 FPS for the S version. While the V100 is a powerful GPU, achieving such high FPS on any device is impressive.

- 🔗 YOLO-World arXiv paper: https://lnkd.in/ddRBKCCX
- 🔗 my YOLO-World technical report: https://blog.roboflow.com/what-is-yolo-world
- 🤗 YOLO-World space: https://huggingface.co/spaces/SkalskiP/YOLO-World

clem adirik samusenps Kukedlc Pretam soulteary Citaman osanseviero DiamanteAmarelo sweat89 MiguelDu rreed-pha victor SkalskiP

SkalskiP DmitryRyumin vladbogo VictorSanh notune osanseviero DiamanteAmarelo MiguelDu


HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition,  Licai Sun  et al.

Repo: https://github.com/sunlicai/hicmae

Paper: https://arxiv.org/abs/2401.05698

wijjjj clem samusenps osanseviero andeliej isa2023 vladbogo fpreiss Suparious Licai


Senku-70B stills undefeated within EQ-Bench, latest updates from the author shows even a further increase in performance, reaching  a new score of 85.09

This new mark outperform some GPT-4 models, closing further the very thin gap between OpenCommunity LLM and Closed source models.

https://huggingface.co/ShinojiResearch/Senku-70B-Full

Ji-Ha eramax clem dillfrescott osanseviero Dlbk fblgit

distantquant clem dillfrescott yingyuan Dlbk nickandbro mlabonne


Happy to share that Filtir, our AI fact-checking pipeline, is now also available as a Hugging Face Space. You can give it a try at: https://huggingface.co/spaces/vladbogo/Filtir.

Feedback is appreciated!

macadeliccc NovoCode osanseviero Squadrongrit leeloolee clem mrfakename kramp

NovoCode osanseviero sayhan clem Jose7juanFdz ioanacroi


At long last, it's been found

The holy grail

The one cable to rule them all

euclaise Hugging-Fox reallygoodtechdeals dsaccvxcvzvzv osanseviero vladbogo yasminemo VictorSanh dillfrescott JuniorDerp


Looks like @Meta thinks we forgot they created PyTorch, so now they've open-sourced Lingua, a powerful and flexible library for training and inferencing large language models.

Things that stand out:

- Architecture: Pure PyTorch `nn.Module` implementation for easy customization.

- Checkpointing: Uses the new PyTorch distributed saving method (.distcp format) for flexible model reloading across different GPU configurations.

- Configuration: Utilizes data classes and YAML files for intuitive setup and modification.

- Profiling: Integrates with xFormers' profiler for automatic MFU and HFU calculation, plus memory profiling.

- Slurm Integration: Includes `stool.py` for seamless job launching on Slurm clusters.

Some results from @Meta to show off:

- 1B parameter models trained on 60B tokens achieve strong performance across various NLP tasks.

- 7B parameter Mamba model (trained on 200B tokens) shows competitive results with Llama 7B on benchmarks like ARC, MMLU, and BBH.

If you're working on LLM research or looking to experiment with cutting-edge language model architectures, Lingua is definitely worth exploring.


Interested in performing inference with an ONNX model?⚡

The Optimum docs about model inference with ONNX Runtime is now much clearer and simpler!

You want to deploy your favorite model on the hub but you don't know how to export it to the ONNX format? You can do it in one line of code as follows:
```py
from optimum.onnxruntime import ORTModelForSequenceClassification

# Load the model from the hub and export it to the ONNX format
model_id = "distilbert-base-uncased-finetuned-sst-2-english"
model = ORTModelForSequenceClassification.from_pretrained(model_id, export=True)
```
Check out the whole guide 👉 https://huggingface.co/docs/optimum/onnxruntime/usage_guides/models

John6666 adeiza-x louisbrulenaudet Aurelien-Morgan itsbrex


Hi HuggingFacers!🤗

As you may have probably heard, in the past weeks three Tech Giants (Microsoft, Amazon and Google) announced that they would bet on nuclear reactors to feed the surging energy demand of data centers, driven by increasing AI data and computational flows.

I try to explain the state of AI energy consumptions, its environmental impact and the key points of "turning AI nuclear" in my last article on HF community blog: https://huggingface.co/blog/as-cle-bert/ai-is-turning-nuclear-a-review

Enjoy the reading!🌱

rmayormartins John6666 den0620 daniel-ltw not-lain louisbrulenaudet SriramRokkam odyss3y julesbsz


Last Thursday at KaggleX organized by Google, I presented a workshop on "Unlocking the Power of Large Language Models (LLMs) for Business Applications" where I explained how we can reduce the size of LLM models to make them more suitable for business use and addressing common resource limitations.
https://drive.google.com/file/d/1p5sT4_DeyBuwCqmYt4dCJKZOgLMpESzR/view


Every adult on the planet knows what a vector is and has the basic understanding of how they are utilized right in their heads. You just don't know it as vector math. You do not know a 2-D vector as a 2-D vector, you know it as a graph. Want to know more? Check out this video, I break down the concept in about 10 minutes and I am positive you will fully understand it by the end: https://youtu.be/Iny2ughcGsA


Artificial Minds, Human Consequences: Unraveling AI’s Impact on Education, Cognition, and Cultural Production

https://empereur-pirate.medium.com/artificial-minds-human-consequences-unraveling-ais-impact-on-education-cognition-and-cultural-a503b88d4524

This text critiques AI language models, particularly in education, arguing their use hinders deep learning, critical thinking, and social-emotional development despite automating tasks. It raises copyright concerns, highlighting lawsuits against companies like OpenAI and Microsoft. AI "creativity" is challenged, emphasizing its reliance on stochastic processes and lack of true understanding. Ethical implications, including bias and misuse, are explored, along with environmental costs. The text contrasts the human brain's complex temporalities and adaptability with the static nature of current AI models, which lack genuine long-term memory and continuous learning.


Made a notable change to the TTS Arena fork. I do not think anyone is interested in which bottomfeeder TTS is better than another beside it. So one of the top 5 TTS is always chosen in a challenge for more scrutiny. Also these top 5 are taken from preliminary results.
https://huggingface.co/spaces/Pendrokar/TTS-Spaces-Arena


SambaNova ☁
⚡ Inference API with cURL Demo: https://huggingface.co/spaces/prithivMLmods/sambanova-inference-api

🔗Sambanova API Documentation : (grab your APIs here)  https://cloud.sambanova.ai/apis

```
export SAMBANOVA_API_KEY=<your token>
```
Sambanova's Inference API.

```
pip install sambanova-gradio
```
SambaNova X Gradio

```
import gradio as gr
import sambanova_gradio

gr.load(
    name='Meta-Llama-3.1-405B-Instruct',
    src=sambanova_gradio.registry,
).launch()
```
📃 Documentation: https://community.sambanova.ai/docs

John6666 victor akhaliq AtAndDev prithivMLmods


I'm now working on finetuning of coding models. If you are GPU-hungry like me, you will find quantized models very helpful. But quantization for finetuning and inference are different and incompatible. So I made two collections here.

Inference (GGUF, via Ollama, CPU is enough)
https://huggingface.co/collections/onekq-ai/ollama-ready-coding-models-67118c3cfa1af2cf04a926d6

Finetuning (Bitsandbytes, QLora, GPU is needed)
https://huggingface.co/collections/onekq-ai/qlora-ready-coding-models-67118771ce001b8f4cf946b2

For quantization, the inference models are far more popular on HF than finetuning models. I use https://huggingface.co/QuantFactory to generate inference models (GGUF), and there are a few other choices.

But there hasn't been such a service for finetuning models. DIY isn't too hard though. I made a few myself and you can find the script in the model cards. If the original model is small enough, you can even do it on a free T4 (available via Google Colab).

If you know a (small) coding model worthy of quantization, please let me know and I'd love to add it to the collections.

djuna John6666 valentimarco RalFinger rajkumar321 regisss


Large Language Models in Targeted Sentiment Analysis (2404.12342)

📢 Excited to share that our studies 📄 "Large Language Models in Targeted Sentiment Analysis for Russian" has recently become in 📘 Springer Lobachevskii Journal of Mathematics 🥳✨ ...

📘 https://link.springer.com/article/10.1134/S1995080224603758

In this studies we provide such a diverse look and experiments over various 🤖 LLM models 🤖 scaled from 7B in two different modes: ❄ zero-shot and 🔥 fine-tuned (Flan-T5 only) using Three-Hop reasoning technique.
We showcase the importance of performing:
💚 text translation into English
💚 application on Chain-of-Thought for Implicit Sentiment Analysis

More:
📄 Arxiv: https://arxiv.org/abs/2404.12342
🧑💻 Code: https://github.com/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework
🤗 Models: https://huggingface.co/papers/2404.12342
🎥 Video @NLPSummit: https://www.youtube.com/watch?v=qawLJsRHzB4

THOR: https://github.com/scofield7419/THOR-ISA


Self-Play Fine-Tuning Converts Weak Language Models to Strong Language
  Models (2401.01335)

Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models

UCLA-AGI has proven that large language models, even weaker large language models, can improve themselves with data only produced by original model. The question they answer in their paper is:

"Can we empower a weak LLM to improve itself without acquiring additional human annotated data?"

They answer this question by the proposal and testing of a novel fine-tuning method they call Self-Play fIne-tuNing (SPIN). The process starts by applying a supervised fine-tune (SFT) to zephyr-7b using all 200k samples of HuggingfaceH4/ultrachat_200k to eliminate the need for a human annotator.

Once the model has completed SFT, the SPIN method suggests generating 50k samples of synthetic data pairs of 'chosen' and 'rejected' samples. The model will be fine tuned on those generations, and this process will repeat for another 3 iterations for a total 200k samples.

This experiment is unique because they propose that their method can yield upwards of 10% performance gains without using any additional human annotated data. The strategy was designed to improve the less strong language models, but with further experimentation could be a formidable strategy for improving language models.

If you would like to explore this strategy for yourself, here are some resources:
Colab: https://colab.research.google.com/drive/1IjDeNVBsRru2-hM_9aauD6gVI-VvJnpk?usp=sharing
Github: https://github.com/uclaml/SPIN
The product of the experiment: https://huggingface.co/UCLA-AGI/zephyr-7b-sft-full-SPIN-iter3
Paper: https://huggingface.co/papers/2401.01335

osanseviero vladbogo Ji-Ha clem samusenps danielus Dlbk julien-c abidlabs RedSparkie


Chain-of-Thought Reasoning Without Prompting (2402.10200)

Chain-of-Thought Reasoning Without Prompting

paper page: https://huggingface.co/papers/2402.10200

In enhancing the reasoning capabilities of large language models (LLMs), prior research primarily focuses on specific prompting techniques such as few-shot or zero-shot chain-of-thought (CoT) prompting. These methods, while effective, often involve manually intensive prompt engineering. Our study takes a novel approach by asking: Can LLMs reason effectively without prompting? Our findings reveal that, intriguingly, CoT reasoning paths can be elicited from pre-trained LLMs by simply altering the decoding process. Rather than conventional greedy decoding, we investigate the top-k alternative tokens, uncovering that CoT paths are frequently inherent in these sequences. This approach not only bypasses the confounders of prompting but also allows us to assess the LLMs' intrinsic reasoning abilities. Moreover, we observe that the presence of a CoT in the decoding path correlates with a higher confidence in the model's decoded answer. This confidence metric effectively differentiates between CoT and non-CoT paths. Extensive empirical studies on various reasoning benchmarks show that the proposed CoT-decoding substantially outperforms the standard greedy decoding.

klebster dagelf dimentox osanseviero victor abdelaziz gsarti sosoai vladbogo alckasoc clem Jose7juanFdz lysandre VaiTon setuc

jeffboudier victor Csplk alckasoc emcon33 clem lysandre moonlock VaiTon


So, I have this idea to (potentially) improve uncertainty quantification for LLM hallucination detection.

The premise is that not all output tokens of a generated response share the same importance. Hallucinations are more dangerous in the form of a noun, date, number, etc.

The idea is to have a "token selection" layer that filters the output token probabilities sequence. Then, we use only the probabilities of the relevant tokens to calculate uncertainty quantification metrics.

The big question is how we know which tokens are the relevant ones. 🤔

My idea is to use the output sequence (decoded one) and use an NLP model (it doesn't need to be a fancy one) to do entity recognition and part-of-speech tagging to the output sequence and then do uncertainty quantification only on the entities that we have set as relevant (nouns, dates, numbers, etc).

What are your thoughts? Have you seen anyone try this before?

Curious to see if anyone has tried this before and know if this would have an impact on the correlation with human-annotated evaluations.

gsarti osanseviero cnmoro clem santiviquez


Premise Order Matters in Reasoning with Large Language Models (2402.08939)

A new paper from Google DeepMind explores the effect of premise ordering on large language models (LLMs) in reasoning tasks. Despite the logical principle that the sequence of premises should not influence the conclusion's validity, the study finds LLMs' performance varies with different premise arrangements. Here's a summary:

The research investigates how the order of premises affects LLMs in logical and mathematical reasoning tasks, challenging the assumption that premise sequence is irrelevant to the outcome.

Key Findings:
* Logical Reasoning: LLMs perform best when premises are in a forward order that aligns with the proof's progression. Deviations from this order result in significant performance drops.
* Mathematical Reasoning: The introduction of the R-GSM benchmark shows a similar sensitivity in LLMs.

Congrats to the authors for their work!

Paper: https://huggingface.co/papers/2402.08939.

osanseviero jjoker999 alielfilali01 flippers clem victor Dlbk cosmojg


🔍 Today's pick in Interpretability & Analysis of LMs: Recovering the Pre-Fine-Tuning Weights of Generative Models by @eliahu, J. Kahana, Y. Hoshen

Using low-rank adapters (LoRA) is nowadays a common practice to fine-tune pre-trained generative models on specific tasks, or align them to human preferences.

This work explores pre-fine tuning weight recovery: given a set of LoRA models with merged weights fine-tuned from the same pre-trained system, the task is to recover the original (unknown) weights of the pre-trained model.

Authors propose SpectralDeTuning, a method framing this task as an optimisation problem alternating a step of approximation for all low-rank tuned matrices using SVD and the closed-form computation of the optimal pre-trained matrix given the approximate low-rank ones.

The LoRA Weight Recovery Attack (LoWRA) benchmark is introduced to evaluate pre-fine tuning weight recovery across language and vision tasks on ViT, Mistral and Stable Diffusion models.

The SpectralDeTuning method is shown to be effective in recovering original models both intrinsically (difference in weights) and behavirally (similar outputs). The main limitations of the approach are the assumption that the rank used by LoRAs is known by the attacker, and the relatively high number of LoRAs needed to provide a good approximation.

📄 Paper: https://huggingface.co/papers/2402.10208

💻 LoWRA Bench: https://huggingface.co/datasets/Eliahu/LoWRA-Bench

🔍 All daily picks in LM interpretability: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9

vladbogo osanseviero sayhan Ajibola Ji-Ha clem Dlbk ssalcedo00


Premise Order Matters in Reasoning with Large Language Models

https://huggingface.co/papers/2402.08939

Large language models (LLMs) have accomplished remarkable reasoning performance in various domains. However, in the domain of reasoning tasks, we discover a frailty: LLMs are surprisingly brittle to the ordering of the premises, despite the fact that such ordering does not alter the underlying task. In particular, we observe that LLMs achieve the best performance when the premise order aligns with the context required in intermediate reasoning steps. For example, in deductive reasoning tasks, presenting the premises in the same order as the ground truth proof in the prompt (as opposed to random ordering) drastically increases the model's accuracy. We first examine the effect of premise ordering on deductive reasoning on a variety of LLMs, and our evaluation shows that permuting the premise order can cause a performance drop of over 30%. In addition, we release the benchmark R-GSM, based on GSM8K, to examine the ordering effect for mathematical problem-solving, and we again observe a significant drop in accuracy, relative to the original GSM8K benchmark.

vladbogo abidlabs AshMariWalker HarshavardhanK clem Dlbk osanseviero ashwinlimaye marinaretik bunnycore alckasoc


SyntaxShap: Syntax-aware Explainability Method for Text Generation (2402.09259)

🔍 Today's pick in Interpretability & Analysis of LMs: SyntaxShap: Syntax-aware Explainability Method for Text Generation by @kamara000, R. Sevastjanova and M. El-Assady

Most model-agnostic post-hoc interpretability methods used nowadays in NLP were originally ported from tabular/CV domains with next to no adjustments to the intrinsic properties of textual inputs.

In this work, authors propose SyntaxSHAP, an adaptation of the Shapely value  approach in which coalitions used to compute marginal contributions to importance scores are constrained by the syntax of the explained sentence. The resulting tree-based coalitions do not satisfy the efficiency assumption of Shapley values but preserves the symmetry, nullity and additivity axioms.

SyntaxSHAP is compared to other model-agnostic approaches on small (GPT-2 117M) and large (Mistral 7B) LMs, showing it produces explanations that are more faithful to model predictions and more semantically meaningful than other common methods, while also being more efficient than the base SHAP method.

📄 Paper: https://huggingface.co/papers/2402.09259

💻 Code: https://github.com/k-amara/syntax-shap

🔍 All daily picks in LM interpretability: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9

lunarflu samusenps clem Dlbk RitaS sayhan


INSIDE: LLMs' Internal States Retain the Power of Hallucination
  Detection (2402.03744)

Eigenvalues to the rescue? 🛟🤔

I found out about this paper thanks to @gsarti 's post from last week; I got curious, so I want to post my take on it. 🤗

The paper proposes a new metric called EigenScore to detect LLM hallucinations. 📄

Their idea is that given an input question, they generate K different answers, take their internal embedding states, calculate a covariance matrix with them, and use it to calculate an EigenScore.

We can think of the EigenScore as the mean of the eigenvalues of the covariance matrix of the embedding space of the K-generated answers.

❓But why eigenvalues?

Well, if the K generations have similar semantics, the sentence embeddings will be highly correlated, and most eigenvalues will be close to 0.

On the other hand, if the LLM hallucinates, the K generations will have diverse semantics, and the eigenvalues will be significantly different from 0.

The idea is pretty neat and shows better results when compared to other methods like sequence probabilities, length-normalized entropy, and other uncertainty quantification-based methods.

💭 What I'm personally missing from the paper is that they don't compare their results with other methods like LLM-Eval and SelfcheckGPT. They do mention that EigenScore is much cheaper to implement than SelfcheckGPT, but that's all on the topic.

Paper: https://huggingface.co/papers/2402.03744

gsarti lunarflu EquinoxElahin samusenps clem zeusfsx linoyts Nikhil7280 Dlbk Parkerlambert123 dillfrescott Citaman shubhamg2208

osanseviero Dlbk gsarti lunarflu dillfrescott Citaman

AnaDP Esquare ashwinlimaye dillfrescott Pent anse10rville


Try out Mixtral 2-bit on a free-tier Google Colab notebook right now!

https://colab.research.google.com/drive/1-xZmBRXT5Fm3Ghn4Mwa2KRypORXb855X?usp=sharing

AQLM method has been recently introduced on transformers main branch

The 2bit model can be found here: https://huggingface.co/BlackSamorez/Mixtral-8x7b-AQLM-2Bit-1x16-hf-test-dispatch

And you can read more about the method here: https://huggingface.co/docs/transformers/main/en/quantization#aqlm

Great work @BlackSamorez and team!

osanseviero alielfilali01 sayhan Chunte victor samusenps macadeliccc clem Dlbk andysalerno jeffboudier loubnabnl linoyts medmac01 mathiasn1 Ji-Ha kramp bombassaro


The Aya project (https://huggingface.co/datasets/CohereForAI/aya_dataset, https://huggingface.co/datasets/CohereForAI/aya_collection and https://huggingface.co/datasets/CohereForAI/aya_evaluation_suite) by CohereForAI got released yesterday ! And today I'am excited to introduce Arabic Aya (2A) 🌟

Arabic Aya is a carefully curated dataset, derived from the vast Aya collection by CohereForAI, tailored specifically for Arabic language processing. It consolidates texts across Modern Standard Arabic (MSA) and other dialects, simplifying access to high-quality data for researchers, developers, and linguists.

🔍 Why Arabic Aya?
- Time-saving : Jump straight into your projects with pre-filtered Arabic texts.
- Diverse applications : Perfect for language modeling, sentiment analysis, dialect identification, and more.
- Community-driven : Your contributions and feedback can help enrich this resource further.

📚 Utilize Arabic Aya for your next NLP/LLM projects and be part of advancing Arabic language technologies. Let’s collaborate to make Arabic AI research more accessible and robust!

Check it out here: https://huggingface.co/datasets/2A2I/Arabic_Aya

vladbogo sayhan samusenps Dlbk ybelkada ajibawa-2023 osanseviero victor clem fqa-cyber kramp ArthurZ abdullah


Lumos : Empowering Multimodal LLMs with Scene Text Recognition (2402.08017)

Meta Reality Labs has developed Lumos, a system that merges Multimodal Large Language Models (MM-LLMs) with Scene Text Recognition (STR) to boost the efficiency of various tasks such as multimodal question-answering and text summarization.

Key aspects of Lumos include:

* Hybrid Computing: Utilizes a combination of on-device and cloud computing to process inputs, aiming to reduce latency.
* STR Components:
  * Region of Interest (ROI) Detection: Focuses on text-rich areas within images for optimized text extraction.
  * Text Detection and Recognition: Ensures high-quality text recognition within the ROI.
  * Reading Order Reconstruction: Arranges recognized text to mimic natural reading order, essential for context understanding.

Lumos demonstrates significant improvement with 80% accuracy in question-answering benchmarks and a low word error rate.

Paper: https://huggingface.co/papers/2402.08017

Congrats to the authors for their work!

Dlbk ybelkada ajibawa-2023 musfiqdehan osanseviero clem Esquare jjoker999 Mrdaffy bunnycore QuantumResearch merve ioanacroi


I think one of the most important ways you can contribute to open-source machine learning in 2024 is through datasets.

On Monday Argilla and Hugging Face launched #data-is-better-together an experiment focused on collectively building datasets on the Hub.

For our V1 experiment we're aiming to collectively rank 50k prompts!

 In the few days since launch we've had:

❤ 158 people contribute
🚀 2,796 prompts ranked

🤔 How Can You Contribute?

1. Sign up if you don’t have a Hugging Face account (why not!?)
2. Go to this Argilla Space and sign in: https://huggingface.co/spaces/DIBT/prompt-collective
3. Read the guidelines and start rating prompts!

You can also join the #data-is-better-together channel in the Hugging Face Discord 🔗 https://discord.com/channels/879548962464493619/1205128865735770142

yjernite samusenps vladbogo TuringsSolutions Dlbk ybelkada osanseviero santiviquez DmitryRyumin kramp julien-c clem dvilasuero pcuenq Sylvestre jeffboudier birgermoell Suparious mathiasn1 alvarobartt NovoCode matlok


NVIDIA released Chat with RTX. Why is it significant? It's NVIDIA's first step towards the vision of "LLM as Operating System" - a locally running, heavily optimized AI assistant that deeply integrates with your file systems, features retrieval as the first-class citizen, preserves privacy, and leverages your desktop gaming GPUs to the full.

A web-hosted version will have a hard time matching the same functionality. NVIDIA is going local even before OpenAI.

https://www.nvidia.com/en-us/ai-on-rtx/chat-with-rtx-generative-ai/

akhaliq victor osanseviero clem mrnemo97 PtiteTartine samusenps KADS-dev vladbogo Dlbk kramp J-Rojas mishig D4ve-R alielfilali01 cncqbaiying manoumhd99 abidlabs Jingya lysandre notune freddyaboulton

samusenps Dlbk mishig clem abidlabs Jingya lysandre notune


🚀Gradio Notebook (beta) is here! A custom component that deploys an interactive notebook UX on your Hugging Face space 🤗

➤ Deploy a Space in minutes without writing any UI code.
➤ Use a combination of text, image, and audio models in a single space
➤ Interact with models in a familiar notebook UI
➤ Shareable URL for model outputs to share with friends

👉 Check out the Space: https://huggingface.co/spaces/lastmileai/gradio-notebook-template
Demo video: https://youtu.be/FlwINB9RmKk
Repo: https://github.com/lastmile-ai/aiconfig/tree/main/gradio-notebook
Docs: https://aiconfig.lastmileai.dev/docs/gradio-notebook

Gradio Notebook was made possible thanks to Gradio 4.0, which introduced the ability for developers to create their own custom components and use them in Gradio apps.

Seeking feedback so please try it out and let us know how to improve the experience, both for space creators and consumers!

tanyarai akhaliq andrewlastmile rholinshead zakariassoul victor Rossdan multimodalart jonathanlastmileai julien-c clem samusenps vigsachi23 vladbogo sayhan ybelkada Dlbk bisnotforbella

akhaliq tanyarai Ankush-LM andrewlastmile rholinshead zakariassoul victor multimodalart julien-c clem vigsachi23 sayhan cnmoro ybelkada kramp Dlbk damerajee

akhaliq tanyarai andrewlastmile zakariassoul multimodalart julien-c clem vigsachi23 sayhan ybelkada Dlbk


BASE TTS: Lessons from building a billion-parameter Text-to-Speech model
  on 100K hours of data (2402.08093)

Amazon presents BASE TTS

Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data

https://huggingface.co/papers/2402.08093

introduce a text-to-speech (TTS) model called BASE TTS, which stands for Big Adaptive Streamable TTS with Emergent abilities. BASE TTS is the largest TTS model to-date, trained on 100K hours of public domain speech data, achieving a new state-of-the-art in speech naturalness. It deploys a 1-billion-parameter autoregressive Transformer that converts raw texts into discrete codes ("speechcodes") followed by a convolution-based decoder which converts these speechcodes into waveforms in an incremental, streamable manner. Further, our speechcodes are built using a novel speech tokenization technique that features speaker ID disentanglement and compression with byte-pair encoding. Echoing the widely-reported "emergent abilities" of large language models when trained on increasing volume of data, we show that BASE TTS variants built with 10K+ hours and 500M+ parameters begin to demonstrate natural prosody on textually complex sentences. We design and share a specialized dataset to measure these emergent abilities for text-to-speech. We showcase state-of-the-art naturalness of BASE TTS by evaluating against baselines that include publicly available large-scale text-to-speech systems: YourTTS, Bark and TortoiseTTS.

vladbogo Dlbk clem AshMariWalker alabulei AdinaY


There's a new leaderboard for vision language models 🤩
The models are ranked based on ELO, you can rate the responses to preselected examples or try with your input 🤗
https://huggingface.co/spaces/WildVision/vision-arena

kadirnar osanseviero YaTharThShaRma999 lunarflu Dlbk ashishkssingh clem samusenps sayhan vincentweisser maj34 sbrandeis


⚔ The TIGERLab's Text2Image arena is here! ⚔
 https://huggingface.co/spaces/TIGER-Lab/GenAI-Arena

Like https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard for LLMs: you prompt, two images emerge, vote for the best one  🏆

With enough votes this will lead to an Elo-based leaderboard for text-to-image models, go vote! 🗳
https://huggingface.co/spaces/TIGER-Lab/GenAI-Arena

osanseviero lunarflu vinesmsuic multimodalart seyf1elislam ldaniel-jmz OmbelineM


🚀 Exciting breakthrough in LLM reliability! 🧠NexusRaven-V2, our cutting-edge function-calling LLM, has set a new standard in minimizing AI hallucinations, surpassing GPT-4's performance in a recent third-party independent research benchmark.

Dive into our latest blog post to explore how we're pioneering reliable agents with minimal hallucinations: https://nexusflow.ai/blogs/towards-reliable-agents-with-minimal-hallucination

Key Highlights:

🏆 Zero Hallucinations: NexusRaven-V2 showcased remarkable accuracy with zero hallucinations in 840 tests, focusing on tool selection and usage – a significant leap over GPT-4 with 23 hallucinations.

📈 Enhanced Success Rates: It boasts a 9% higher success rate than GPT-4 in information-seeking applications requiring meticulous attention to detail and a 4% increase in adversarial scenarios that demand a deep understanding of tool documentation, even with vague tool and API argument names.

Try NexusRaven-V2 on Huggingface: https://huggingface.co/Nexusflow/NexusRaven-V2-13B

Check out the original third-party benchmark: https://arxiv.org/abs/2401.08326

osanseviero venkat-srinivasan-nexusflow brian-yu-nexusflow samusenps victor lvwerra Dlbk bunnycore fffiloni lewtun not-lain

osanseviero macadeliccc venkat-srinivasan-nexusflow brian-yu-nexusflow francis123 victor lvwerra Dlbk marinaretik jian-zhang-nexusflow

samusenps francis123 osanseviero lvwerra Dlbk tomaarsen not-lain tuantm jian-zhang-nexusflow sbrandeis

Dlbk ydeng9 not-lain jian-zhang-nexusflow


OS-Copilot: Towards Generalist Computer Agents with Self-Improvement (2402.07456)

OS-Copilot is a new framework for creating computer agents such as FRIDAY. This framework enables agents to interact seamlessly with your operating system, handling tasks like file management, multimedia editing, and more.

The system has three components:
* Planner: It takes complex user requests and breaks them down into manageable subtasks for efficient execution.
* Configurator: It prepares tasks for execution based on your preferences and available commands using a memory mechanism.
* Actor: It executes the tasks and learns from feedback, ensuring continuous improvement.

FRIDAY outperforms other methods on GAIA, a comprehensive benchmark. To answer the questions from GAIA, the agents need skills to calculate numbers, browse the web, process video and speech signal and others.

Resources:
* Paper: https://huggingface.co/papers/2402.07456
* Project GitHub: https://github.com/OS-Copilot/FRIDAY
* Project page: https://os-copilot.github.io/

Congrats to the authors Wu, Zhiyong et al. for their work!

clem macadeliccc Yamazaki-Hakubi victor sayhan dbalda Dlbk marinaretik Moibe


Fine-tune 7B models on free-tier Colab hardware using Unsloth 🦥

Unsloth is a framework for fine tuning language models boasting a 0% loss in accuracy  while using no approximation methods. They offer  a trainer for both supervised fine-tuning (SFT) and direct preference optimization (DPO) that can increase speed of fine-tuning by up to 5x.
This is achieved by adding LoRa adapters. This way they only need to train 1 to 10% of the total parameters. You can export a LoRa adapter or merge to 16-bit for a full finetune. The resulting model is prepared for use in vLLM for faster inference.

Additionally, Huggingface has integrated Unsloth into the documentation for DPO training and reported 18.6% performance gains on T4.

This sets a new standard for fine-tuning large language models. If you would like to explore this methodology for yourself I have provided a notebook "AutoSloth," where you can fine tune using either SFT or DPO and it will upload to HF with a prefilled Unsloth README 🦥 and a Q8_0 quantization.

The SFT example is set up for free tier usage, but the DPO example is set up for an A100. The DPO example can be altered to work on T4 but I wanted to include more than one example.

Colab Stats during training:
+ Model: unsloth/mistral-7b-bnb-4bit
+ Dataset: yahma/alpaca-cleaned
+ Batch size: 2
+ Gradient steps: 4
+ System RAM: 8.5 / 51.0 GB
+ VRAM (T4): 13.6 / 15.0 GB

Resources:
🦥Unsloth: https://github.com/unslothai/unsloth
🦥AutoSloth: https://colab.research.google.com/drive/1Zo0sVEb2lqdsUm9dy2PTzGySxdF9CNkc?usp=sharing
🤗HF-Unsloth-docs: https://huggingface.co/docs/trl/main/en/dpo_trainer#accelerate-dpo-fine-tuning-using-unsloth
🤗HF-Unsloth Blog Post: https://huggingface.co/blog/unsloth-trl

alielfilali01 clem sinanyuksel eramax samusenps osanseviero victor vladbogo ybelkada alvarobartt koalazf99 tomaarsen Citaman Dlbk davanstrien kolaposki julien-c Ji-Ha maywell Lovnir Krasi pandora-s boapps felicityworld pcuenq horiguchidotconf

ybelkada Dlbk julien-c felicityworld horiguchidotconf


@clem ist das der erste nicht Englische post auf huggingface?👋🏽 🇩🇪🇫🇷🇮🇹🇪🇸🇮🇳…

julien-c osanseviero Goofy-24 clem Fredithefish


Buffer Overflow in Mixture of Experts (2402.05526)

Mixture of experts: beware 🛡⚔

New paper by DeepMind: Buffer Overflow in MoE https://huggingface.co/papers/2402.05526

The paper shows an adversarial attack strategy in which a user sends malicious queries that can affect the output of other user queries from the same batch.

So if in the same batch we have
- User A benign query
- User B malicious query
The response for A might be altered!😱

How is this possible?
One approach is to fill the token buffers with adversarial data, hence forcing the gating to use the non-ideal experts or to entirely drop the bening tokens (in the case of finite limit size).

This assumes that the adversary can use the model as a black-box but can observe the logit outputs + ensure that the data is always grouped in the same batch.

How to mitigate this?
- Randomize batch order (and even run twice if some queries are very sensitive)
- Use a large capacity slack
- Sample from gate weights instead of top-k (not great IMO, as that require more memory for inference)

Very cool paper!!

NickyNicky julien-c macadeliccc ivanfioravanti clem samusenps victor JosephGesnouin pratjz blancsw marinaretik juancopi81 Dlbk djmcflush EmelMk surfiend Yasbok 0xe69756

macadeliccc clem samusenps JosephGesnouin blancsw santiviquez Dlbk abunchofrandomwords 0xe69756

meefs blancsw davanstrien Dlbk Yhyu13 Epiculous


SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step
  Reasoning (2308.00436)

Hey GPT, check yourself...

Here is a black-box method for hallucination detection that shows strong correlation with human annotations. 🔥

💡 The idea is the following: ask GPT, or any other powerful LLM, to sample multiple answers for the same prompt, and then ask it if these answers align with the statements in the original output. Make it say yes/no and measure the frequency with which the generated samples support the original statements.

This method is called SelfCheckGPT with Prompt and shows very nice results. 👀

The downside, we have to do many LLM calls just to evaluate a single generated paragraph... 🙃

More details and variations of this method are in the paper: https://huggingface.co/papers/2308.00436

arora-anmol osanseviero clem samusenps victor pulkitmehtawork Gvangelista yjernite


OS-Copilot

Towards Generalist Computer Agents with Self-Improvement

https://huggingface.co/papers/2402.07456

Autonomous interaction with the computer has been a longstanding challenge with great potential, and the recent proliferation of large language models (LLMs) has markedly accelerated progress in building digital agents. However, most of these agents are designed to interact with a narrow domain, such as a specific software or website. This narrow focus constrains their applicability for general computer tasks. To this end, we introduce OS-Copilot, a framework to build generalist agents capable of interfacing with comprehensive elements in an operating system (OS), including the web, code terminals, files, multimedia, and various third-party applications. We use OS-Copilot to create FRIDAY, a self-improving embodied agent for automating general computer tasks. On GAIA, a general AI assistants benchmark, FRIDAY outperforms previous methods by 35%, showcasing strong generalization to unseen applications via accumulated skills from previous tasks. We also present numerical and quantitative evidence that FRIDAY learns to control and self-improve on Excel and Powerpoint with minimal supervision. Our OS-Copilot framework and empirical findings provide infrastructure and insights for future research toward more capable and general-purpose computer agents.

morgan osanseviero clem samusenps angkul07 mindrage truong-xuan-linh impactframes


Fine-tuning LLMs is rad, but how do you manage all your checkpoints and evals in a production setting?

We partnered with @hamel to ship an Enterprise Model Management course packed full of learnings for those training, evaluating and deploying models at work.

Topics include:
- What webhooks are & how to use them to create integrations with different tools
- How to automate train -> eval runs
- Improving model governance and documentation
- Comparing candidate and baseline models
- Design patterns & recipes
- Lots more...

Would love to hear what you think!

👉  https://www.wandb.courses/courses/enterprise-model-management

osanseviero altryne macadeliccc clem samusenps


Show Me How It's Done: The Role of Explanations in Fine-Tuning Language
  Models (2402.07543)

🔍 Today's pick in Interpretability & Analysis of LMs: Show Me How It's Done: The Role of Explanations in Fine-Tuning Language Models by M. Ballout @krumnack et al.

Authors propose a fine-tuning procedure in which a classification task is framed as generation and augmented with a natural language explanation to clarify intermediate reasoning steps. The procedure is applied to fine-tune language models of various sizes on the ListOps dataset, containing synthetically-generated instructions on sequences of numbers.

Authors find that explanations contribute to improving model performances across all tested model sizes and explanations lengths. Smaller language models appear to benefit the most from this approach in terms of convergence speed, performance and input length generalisation, especially when given more exhaustive explanations.

📄 Paper: https://huggingface.co/papers/2402.07543

💻 Code: https://github.com/BalloutAI/Fine-tuning-with-Explanation

🔍 All daily picks in LM interpretability: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9


I love the new `Viewer` and i didn't knew how much i needed it until now
@sylvain , @lhoestq  and team, GREAT JOB 🔥 and THANK YOU 🤗

davanstrien osanseviero julien-c macadeliccc clem DamarJati kramp