Backward Lens: Projecting Language Model Gradients into the Vocabulary
  Space (2402.12865)

🔍 Today's pick in Interpretability & Analysis of LMs: Backward Lens: Projecting Language Model Gradients into the Vocabulary Space by @shaharkatz @belinkov @mega @liorwolf

Recent interpretability works explore intermediate model representations by projecting them to vocabulary space. This work explores projecting gradients computed from the backward pass to vocabulary space to explain how a single forward-backward pass edits LM knowledge.

Authors identify a mechanism they dub “imprint and shift” in the forward module in transformer layer. Specifically, the “imprint” refers to the first layer, to or from which the learning process adds or subtracts copies of the intermediate inputs encountered during the forward pass. The “shift” refers to the second matrix, where the weights are shifted by the embedding of the target token.

Authors note that the dominant components in constructing gradients are derived from the outer product of the last token’s input and the Vector-Jacobian Product, and that the latter contains the embedding of the target token.

In light of this, a new editing approach named “forward pass shifting” is proposed to update the shifting component of a layer’s feedforward module without backpropagation, using only layer inputs and target token embeddings. The method performs on par with significantly more expensive editing approaches like ROME for single-fact editing, but is less robust to paraphrasing.

Authors note that these results provide promising evidence on the possibility of finding shortcuts to fine-tuning by directly injecting knowledge in model layers.

📄 Paper: https://huggingface.co/papers/2402.12865

🔍 All daily picks in LM interpretability: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9


FasterViT: Fast Vision Transformers with Hierarchical Attention (2306.06189)

🚀🔥🌟 New Research Alert - ICLR 2024! 🌟🔥🚀
📄 Title: FasterViT: Fast Vision Transformers with Hierarchical Attention

👥 Authors: @ahatamiz, @slivorezzz et al.

📅 Conference: ICLR, May 7-11, 2024 | Vienna, Austria 🇦🇹

🔗 Paper: https://huggingface.co/papers/2306.06189

🔗 Model 🤖 : https://huggingface.co/nvidia/FasterViT
🔗 Repo: https://github.com/NVlabs/FasterViT

📚 More Papers: more cutting-edge research presented at other conferences in the https://huggingface.co/spaces/DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin

🔍 Keywords: #VisionTransformers #DeepLearning #ComputerVision #ICLR2024 #MachineLearning #HierarchicalAttention #NeuralNetworks #Research #ArtificialIntelligence #Innovation

osanseviero merve Alexandro14 clem ahatamiz chansung Falah dillfrescott


REALIGN is a new method designed to improve the alignment of Large Language Models (LLMs) with human values by reformatting instruction data. This approach enhances LLM performance across various metrics by aligning responses with predefined criteria and evidence.

Key points:

* REALIGN has three steps: criteria definition, retrieval augmentation, and response reformatting
* It rewrites pairs (query, response) to enhance data quality for fine-tuning LLMs.
* It has shown significant improvements in general alignment, math reasoning and other tasks.

Congrats to the authors for their work!

Paper: https://huggingface.co/papers/2402.12219
Code: https://github.com/GAIR-NLP/ReAlign


MANIFESTO
After working in fashion e-commerce for years I've come to the conclusion that in e-commerce we do not sell clothes... we sell images of clothes. Compressed, digital versions of physical products. As Roland Barthes pointed out in The Fashion System, a product image is a symbol or metaphor of a product. Media--in this case images--mediates the space between customer and product; viewer and object. Images can be altered, changed, corrupted, photoshopped, edited, deleted, or imagined. E-commerce products (or e-commerce photos) can thought of as a possibility space of digital pixels. AI/ML can analyze, manipulate, and create within this "possibility space of pixels"--thus it can be observed that there are opportunities to intervene in the physical fashion world through the imagination of artificial intelligence. Not to replace human creativity--but to augment it. To make it ART-ificial. Art is an artificial representation of reality. AI images are an artificial representation of reality. The sewing  machine greatly increased the efficiency of clothing production. Similarly, AI has greatly increased efficiency image production, in our case product photo production. The fashion design paradigm of the past century (design->produce->photograph) has been flipped on this head. Instead of going from physical clothing to digital image via photography--we can go from digital image to physical clothing via stable diffusion. We are writing the chapter of Understanding Media that Marshall McLuhan never imagined. Virtual production hasn't replaced the physical production; it has simply made it out of style.

fffiloni TuringsSolutions vladbogo dcsdcsdasdasa osanseviero Taf2023 aidystark kramp samusenps angelium111 Madisen Wauplin lunarflu pascalnjue Chieh-Yun sghosh2 DocRace ozgung Krishana671 Jia-ao Nymbo strangebose2 marinarosa MustardWhale516

varadpuntambekar strangebose2 Nymbo marinarosa

Максим Николаевич Подъяпольский


Fine-grained Hallucination Detection and Editing for Language Models (2401.06855)

Fantastic Beasts (*Hallucinations*) and Where to Find Them 🔎🧌

This paper breaks down LLM hallucinations into six different types:

1⃣ Entity: Involves errors in nouns. Changing that single entity can make the sentence correct.

2⃣ Relation: Involves errors in verbs, prepositions, or adjectives. They can be fixed by correcting the relation.

3⃣ Contradictory: Sentences that contradict factually correct information.

4⃣ Invented: When the LLM generates sentences with concepts that don't exist in the real world.

5⃣ Subjective: When the LLM generates sentences influenced by personal beliefs, feelings, biases, etc.

6⃣ Unverifiable: When the LLM comes up with sentences containing information that can't be verified. E.g., Personal or private matters.

The first two types of hallucinations are relatively easy to correct, given that we can rewrite them by changing the entity or relation. However, the other four would mostly need to be removed to make the sentence factually correct.

Paper:  https://huggingface.co/papers/2401.06855

Kukedlc vladbogo bisnotforbella osanseviero jeffboudier andrewrreed EpsilonPhoenix7 loudog2323 Hanyu66


ICYMI! Nomic Embed v1.5: Resizable Production Embeddings with Matryoshka Representation Learning

- Variable embedding dimension from 64 <-> 768
- Outperforms text-embedding-ada-002 while achieving a 3x memory reduction
- Day 1 integrations with Langchain, LlamaIndex, MongoDB, and Sentence Transformers

Check out
https://huggingface.co/nomic-ai/nomic-embed-text-v1.5 for the model weights.

Technical report: https://static.nomic.ai/reports/2024_Nomic_Embed_Text_Technical_Report.pdf
Blog Post: https://blog.nomic.ai/posts/nomic-embed-matryoshka
Original Tweet Thread: https://x.com/nomic_ai/status/1757782157374734665?s=20

philschmid Xenova osanseviero Taf2023 davanstrien sayhan linhphanff Yhyu13


Working through the Reddit dataset, one thing that occurs to me is we pretty much always train LLMs to be a conversation between 2 parties like Bot/Human or Instruction/Response.

It seems far more common with internet data that we have multi-speaker/group discussions with a dynamic number of speakers. This also seems to be more realistic to the real world too and requires a bit more understanding to model.

Is there some research into this? I have some ideas of how I'd like to implement it, but I wonder if some work has already been done here?

yungwarlock Moooritz krutarth53 pranav-deshpande ClaudioItaly Stephenmcgurrin sthakur2 poetofgrindset Noomam Luxter77


Speculative Streaming: Fast LLM Inference without Auxiliary Models (2402.11131)

Speculative Streaming

Fast LLM Inference without Auxiliary Models

https://huggingface.co/papers/2402.11131

Speculative decoding is a prominent technique to speed up the inference of a large target language model based on predictions of an auxiliary draft model. While effective, in application-specific settings, it often involves fine-tuning both draft and target models to achieve high acceptance rates. As the number of downstream tasks grows, these draft models add significant complexity to inference systems. We propose Speculative Streaming, a single-model speculative decoding method that fuses drafting into the target model by changing the fine-tuning objective from next token prediction to future n-gram prediction. Speculative Streaming speeds up decoding by 1.8 - 3.1X in a diverse set of tasks, such as Summarization, Structured Queries, and Meaning Representation, without sacrificing generation quality. Additionally, Speculative Streaming is parameter-efficient. It achieves on-par/higher speed-ups than Medusa-style architectures while using ~10000X fewer extra parameters, making it well-suited for resource-constrained devices.


I'm working on a templated Quarto website and was looking for some good placeholder content. Do you have any ideas? The ideal content would be:
- Widely legible
- Short
- Include long form text
- Would benefit from some interactive applications


We fine-tuned 25 mistral-7b models that outperform GPT-4 on task-specific use cases!

Check them out at LoRA Land: https://pbase.ai/3uFh7Qc

You can prompt them and compare their results to mistral-7b-instruct in real time!

They're also on HF for you to play with: https://huggingface.co/predibase

Let us know what you think!

Kukedlc clem arnavgrg samusenps gsarti cnmoro anthonymikinka chandlerTSLabs lunarflu Dlbk dillfrescott mathiasn1 steventrouble

9bow kramp Pent chandlerTSLabs dillfrescott


🔥 New LLM leaderboard blog: Open Ko LLM!

One of the oldest leaderboards on the hub, it has already evaluated more than 1000 models! It uses Korean translations of MMLU, ARC, HellaSwag, TruthfulQA, and a new dataset, Korean CommonGen, about specific common sense alignement.

https://huggingface.co/spaces/upstage/open-ko-llm-leaderboard

What's interesting about this leaderboard is how it drove LLM development in Korea, with on average about 4 submissions/models per day since it started!
Really looking forward to seeing similar initiatives in other languages, to help qualitative models emerge outside of "just English" (for the other 2/3rds of the world).

Read more about how the leaderboard in the intro blog: https://huggingface.co/blog/leaderboards-on-the-hub-upstage
Congrats to @Chanjun , @hunkim and the Upstage team!

Chanjun kramp KnutJaegersberg clem vladbogo jeffboudier sbrandeis


In-Context Learning Demonstration Selection via Influence Analysis (2402.11750)

🔍 Today's pick in Interpretability & Analysis of LMs: In-Context Learning Demonstration Selection via Influence Analysis
 by Vinay M.S. @minhhaovan X. Wu

Recent work showed how the performance of LMs using in-context learning (ICL) is heavily dependent on selected demonstrations.

This work introduces InfICL, a demonstration selection method using influence functions to identify salient training examples to use as demonstrations at inference time. InfICL is tested alongside other examples selection baselines for prompting medium-sized LLMs for COLA and RTE, showing improvements over other methods especially when a smaller number of in-context examples is used.

📄 Paper: https://huggingface.co/papers/2402.11750

🔍 All daily picks in LM interpretability: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9

kramp clem vladbogo santiviquez nickandbro


📢 New Research Alert - AAAI 2024!
📄 Title: CARAT: Contrastive Feature Reconstruction and Aggregation for Multi-Modal Multi-Label Emotion Recognition

👥 Authors: Cheng Peng et al.

📅 Conference: AAAI, February 20-27, 2024 | Vancouver, Canada 🇨🇦

🔗 Paper: https://arxiv.org/abs/2312.10201

🔗 Repository: https://github.com/chengzju/CARAT

📚 More Papers: Explore a collection of exciting papers presented at AAAI 2024 and other conferences in the repositories:
- AAAI 2024 Papers: https://github.com/DmitryRyumin/AAAI-2024-Papers, @DmitryRyumin
- Other Conferences: https://huggingface.co/spaces/DmitryRyumin/NewEraAI-Papers

🔍 Keywords: #EmotionRecognition #MultiModal #AI #Research #AAAI2024 #MachineLearning


Isn't it sad that VLMs don't have any inference parameters for the vision part? Well, MC-LLaVA now has two whole knobs you can use to make it find even the smallest details! I finally (almost) properly implemented multi-crop, and now you can control the number of crops and how many image tokens will be generated. The video shows how, by increasing the number of crops and tokens, my 3B model correctly identifies the 30x90 pixel logo in the 3200x3000 pixel image.
Other notable updates:
- I use SigLIP from Transformers, so you don't need to install additional libraries.
- the model now supports auto classes, so you can create the model and processor with only two lines.
- performance increased by 10%+ across all benchmarks.

The work is far from over, but it feels like good progress.

The model on the hub: https://huggingface.co/visheratin/MC-LLaVA-3b
You can try the model here: https://huggingface.co/spaces/visheratin/mc-llava-3b

samusenps osanseviero Sylvestre lusstta kinetical dillfrescott Azamat1k Alexandro14

samusenps clem lusstta okbottom dillfrescott


Hi, welcome to my first post here!

I am slowly wrangling about 5 years of reddit comments (2015-2020). It's a total of billions samples that can be filtered as comment-reply pairs, chains of discussion, filtered by subreddit, up/down votes, controversy, sentiment, and more.

Any requests or ideas for curated datasets from here? I'll also tinker with uploading the entire dataset potentially in chunks or something, but it's quite a few terabytes in total, so I'll need to break it up still. I have some ideas for datasets I personally want too, but curious if anyone has something they'd really like to see that sounds interesting too.

osanseviero Dewa clem VictorSanh kramp mrfakename samusenps Xenova qnguyen3 gsarti alielfilali01 MarinaraSpaghetti Noomam

osanseviero Dewa clem isavita samusenps ivanfioravanti


🚀 Exciting news from 🤗 PEFT!

We are introducing new merging methods for LoRA adapters. These methods allow for retaining the unique capabilities of individual LoRAs while enabling them to combine their strengths: https://huggingface.co/blog/peft_merging

We explored the application of merging LoRA adapters in the context of personal code copilot before 🚀👾✨. Please go through the below thread on it:  https://x.com/sourab_m/status/1718008115726283004?s=20

New merging methods `ties`, `dare`, and `magnitude_prune` introduced alongside existing methods `cat`, `linear`, and `svd`. Blogpost details each method. These methods can be applied on-the-fly during inference time instead of merging offline enabling great developer UX. ✨

How do I merge my LoRA adapters?
Easy, use class method `add_weighted_adapter()`. For example, below you can see how we can combine three LoRA adapters using `ties` method. We can observe that merged adapter can retain the capabilities of individual adapters!

Now that we have seen they can retain individual LoRAs, how about use cases wherein we require the capabilities from multiple LoRAs being merged/combined?  Below is an application of it in text-to-image domain. 🖼

Kudos to @prateeky2806 (TIES author) and Le Yu (DARE author) for their kind and generous guidance on the PRs! Also, if you want to explore full model merging, refer to super cool projects like https://github.com/arcee-ai/mergekit/tree/main,  https://github.com/Gryphe/BlockMerge_Gradient and https://github.com/yule-BUAA/MergeLM/tree/main.

Excited to see what the community creates on top of this! 🚀✨ #LetsBuildTogether

jeffboudier samusenps clem ybelkada Kukedlc boapps vWake SnehShah impactframes Citaman MexIvanov ClayFace vanillaOVO madstuntman11 bachvudinh

kramp lysandre osanseviero vladbogo samusenps clem ybelkada impactframes madstuntman11


In Search of Needles in a 10M Haystack

Recurrent Memory Finds What LLMs Miss

paper addresses the challenge of processing long documents using generative transformer models. To evaluate different approaches, we introduce BABILong, a new benchmark designed to assess model capabilities in extracting and processing distributed facts within extensive texts. Our evaluation, which includes benchmarks for GPT-4 and RAG, reveals that common methods are effective only for sequences up to 10^4 elements. In contrast, fine-tuning GPT-2 with recurrent memory augmentations enables it to handle tasks involving up to 10^7 elements. This achievement marks a substantial leap, as it is by far the longest input processed by any open neural network model to date, demonstrating a significant improvement in the processing capabilities for long sequences


Recovering the Pre-Fine-Tuning Weights of Generative Models (2402.10208)

Spectral DeTuning is a new method that successfully recovers the original weights of generative models before they were fine-tuned with human feedback or other customization. It shows that pre-fine-tuning weights are recoverable and methods fine-tuned with LoRA can be susceptible to a new weight recovery type of attack.

Key aspects of the paper:
	•	It introduces Spectral DeTuning for reversing Low-rank Adaptation (LoRA) fine-tuning, targeting original weight restoration through spectral analysis.
	•	LoWRA Bench Dataset: It introduces a dataset for testing Spectral DeTuning across various models and tasks, featuring extensive layers for comprehensive evaluation.
	•	It reveals LoRA fine-tuned models' vulnerability to weight recovery attacks, questioning the security of fine-tuning modifications.

Congrats to the authors for their work!

Paper: https://huggingface.co/papers/2402.10208
Dataset: https://huggingface.co/datasets/Eliahu/LoWRA-Bench
Project page: https://vision.huji.ac.il/spectral_detuning/
Code: https://github.com/eliahuhorwitz/Spectral-DeTuning

osanseviero PeepDaSlan9 samusenps clem deepkyu Svngoku sayhan ababio


YOLO-World: Real-Time, Zero-Shot Object Detection 🔥 🔥 🔥

YOLO-World was designed to solve a limitation of existing zero-shot object detection models: speed. Whereas other state-of-the-art models use Transformers, a powerful but typically slower architecture, YOLO-World uses the faster CNN-based YOLO architecture.

YOLO-World provides three models: small with 13M (re-parametrized 77M), medium with 29M (re-parametrized 92M), and large with 48M (re-parametrized 110M) parameters.

The YOLO-World team benchmarked the model on the LVIS dataset and measured their performance on the V100 without any performance acceleration mechanisms like quantization or TensorRT.

According to the paper, YOLO-World reached 35.4 AP with 52.0 FPS for the L version and 26.2 AP with 74.1 FPS for the S version. While the V100 is a powerful GPU, achieving such high FPS on any device is impressive.

- 🔗 YOLO-World arXiv paper: https://lnkd.in/ddRBKCCX
- 🔗 my YOLO-World technical report: https://blog.roboflow.com/what-is-yolo-world
- 🤗 YOLO-World space: https://huggingface.co/spaces/SkalskiP/YOLO-World

clem adirik samusenps Kukedlc Pretam soulteary Citaman osanseviero DiamanteAmarelo sweat89 MiguelDu rreed-pha victor SkalskiP

SkalskiP DmitryRyumin vladbogo VictorSanh notune osanseviero DiamanteAmarelo MiguelDu


HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition,  Licai Sun  et al.

Repo: https://github.com/sunlicai/hicmae

Paper: https://arxiv.org/abs/2401.05698

wijjjj clem samusenps osanseviero andeliej isa2023 vladbogo fpreiss Suparious Licai


Senku-70B stills undefeated within EQ-Bench, latest updates from the author shows even a further increase in performance, reaching  a new score of 85.09

This new mark outperform some GPT-4 models, closing further the very thin gap between OpenCommunity LLM and Closed source models.

https://huggingface.co/ShinojiResearch/Senku-70B-Full

Ji-Ha eramax clem dillfrescott osanseviero Dlbk fblgit

distantquant clem dillfrescott yingyuan Dlbk nickandbro mlabonne


Happy to share that Filtir, our AI fact-checking pipeline, is now also available as a Hugging Face Space. You can give it a try at: https://huggingface.co/spaces/vladbogo/Filtir.

Feedback is appreciated!

macadeliccc NovoCode osanseviero Squadrongrit leeloolee clem mrfakename kramp

NovoCode osanseviero sayhan clem Jose7juanFdz ioanacroi


At long last, it's been found

The holy grail

The one cable to rule them all

euclaise Hugging-Fox reallygoodtechdeals dsaccvxcvzvzv osanseviero vladbogo yasminemo VictorSanh dillfrescott JuniorDerp


Self-Play Fine-Tuning Converts Weak Language Models to Strong Language
  Models (2401.01335)

Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models

UCLA-AGI has proven that large language models, even weaker large language models, can improve themselves with data only produced by original model. The question they answer in their paper is:

"Can we empower a weak LLM to improve itself without acquiring additional human annotated data?"

They answer this question by the proposal and testing of a novel fine-tuning method they call Self-Play fIne-tuNing (SPIN). The process starts by applying a supervised fine-tune (SFT) to zephyr-7b using all 200k samples of HuggingfaceH4/ultrachat_200k to eliminate the need for a human annotator.

Once the model has completed SFT, the SPIN method suggests generating 50k samples of synthetic data pairs of 'chosen' and 'rejected' samples. The model will be fine tuned on those generations, and this process will repeat for another 3 iterations for a total 200k samples.

This experiment is unique because they propose that their method can yield upwards of 10% performance gains without using any additional human annotated data. The strategy was designed to improve the less strong language models, but with further experimentation could be a formidable strategy for improving language models.

If you would like to explore this strategy for yourself, here are some resources:
Colab: https://colab.research.google.com/drive/1IjDeNVBsRru2-hM_9aauD6gVI-VvJnpk?usp=sharing
Github: https://github.com/uclaml/SPIN
The product of the experiment: https://huggingface.co/UCLA-AGI/zephyr-7b-sft-full-SPIN-iter3
Paper: https://huggingface.co/papers/2401.01335

osanseviero vladbogo Ji-Ha clem samusenps danielus Dlbk julien-c abidlabs RedSparkie


Chain-of-Thought Reasoning Without Prompting (2402.10200)

Chain-of-Thought Reasoning Without Prompting

paper page: https://huggingface.co/papers/2402.10200

In enhancing the reasoning capabilities of large language models (LLMs), prior research primarily focuses on specific prompting techniques such as few-shot or zero-shot chain-of-thought (CoT) prompting. These methods, while effective, often involve manually intensive prompt engineering. Our study takes a novel approach by asking: Can LLMs reason effectively without prompting? Our findings reveal that, intriguingly, CoT reasoning paths can be elicited from pre-trained LLMs by simply altering the decoding process. Rather than conventional greedy decoding, we investigate the top-k alternative tokens, uncovering that CoT paths are frequently inherent in these sequences. This approach not only bypasses the confounders of prompting but also allows us to assess the LLMs' intrinsic reasoning abilities. Moreover, we observe that the presence of a CoT in the decoding path correlates with a higher confidence in the model's decoded answer. This confidence metric effectively differentiates between CoT and non-CoT paths. Extensive empirical studies on various reasoning benchmarks show that the proposed CoT-decoding substantially outperforms the standard greedy decoding.

klebster dagelf dimentox osanseviero victor abdelaziz gsarti sosoai vladbogo alckasoc clem Jose7juanFdz lysandre VaiTon setuc

jeffboudier victor Csplk alckasoc emcon33 clem lysandre moonlock VaiTon


So, I have this idea to (potentially) improve uncertainty quantification for LLM hallucination detection.

The premise is that not all output tokens of a generated response share the same importance. Hallucinations are more dangerous in the form of a noun, date, number, etc.

The idea is to have a "token selection" layer that filters the output token probabilities sequence. Then, we use only the probabilities of the relevant tokens to calculate uncertainty quantification metrics.

The big question is how we know which tokens are the relevant ones. 🤔

My idea is to use the output sequence (decoded one) and use an NLP model (it doesn't need to be a fancy one) to do entity recognition and part-of-speech tagging to the output sequence and then do uncertainty quantification only on the entities that we have set as relevant (nouns, dates, numbers, etc).

What are your thoughts? Have you seen anyone try this before?

Curious to see if anyone has tried this before and know if this would have an impact on the correlation with human-annotated evaluations.

gsarti osanseviero cnmoro clem santiviquez


Premise Order Matters in Reasoning with Large Language Models (2402.08939)

A new paper from Google DeepMind explores the effect of premise ordering on large language models (LLMs) in reasoning tasks. Despite the logical principle that the sequence of premises should not influence the conclusion's validity, the study finds LLMs' performance varies with different premise arrangements. Here's a summary:

The research investigates how the order of premises affects LLMs in logical and mathematical reasoning tasks, challenging the assumption that premise sequence is irrelevant to the outcome.

Key Findings:
* Logical Reasoning: LLMs perform best when premises are in a forward order that aligns with the proof's progression. Deviations from this order result in significant performance drops.
* Mathematical Reasoning: The introduction of the R-GSM benchmark shows a similar sensitivity in LLMs.

Congrats to the authors for their work!

Paper: https://huggingface.co/papers/2402.08939.

osanseviero jjoker999 alielfilali01 flippers clem victor Dlbk cosmojg


🔍 Today's pick in Interpretability & Analysis of LMs: Recovering the Pre-Fine-Tuning Weights of Generative Models by @eliahu, J. Kahana, Y. Hoshen

Using low-rank adapters (LoRA) is nowadays a common practice to fine-tune pre-trained generative models on specific tasks, or align them to human preferences.

This work explores pre-fine tuning weight recovery: given a set of LoRA models with merged weights fine-tuned from the same pre-trained system, the task is to recover the original (unknown) weights of the pre-trained model.

Authors propose SpectralDeTuning, a method framing this task as an optimisation problem alternating a step of approximation for all low-rank tuned matrices using SVD and the closed-form computation of the optimal pre-trained matrix given the approximate low-rank ones.

The LoRA Weight Recovery Attack (LoWRA) benchmark is introduced to evaluate pre-fine tuning weight recovery across language and vision tasks on ViT, Mistral and Stable Diffusion models.

The SpectralDeTuning method is shown to be effective in recovering original models both intrinsically (difference in weights) and behavirally (similar outputs). The main limitations of the approach are the assumption that the rank used by LoRAs is known by the attacker, and the relatively high number of LoRAs needed to provide a good approximation.

📄 Paper: https://huggingface.co/papers/2402.10208

💻 LoWRA Bench: https://huggingface.co/datasets/Eliahu/LoWRA-Bench

🔍 All daily picks in LM interpretability: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9

vladbogo osanseviero sayhan Ajibola Ji-Ha clem Dlbk ssalcedo00


Premise Order Matters in Reasoning with Large Language Models

https://huggingface.co/papers/2402.08939

Large language models (LLMs) have accomplished remarkable reasoning performance in various domains. However, in the domain of reasoning tasks, we discover a frailty: LLMs are surprisingly brittle to the ordering of the premises, despite the fact that such ordering does not alter the underlying task. In particular, we observe that LLMs achieve the best performance when the premise order aligns with the context required in intermediate reasoning steps. For example, in deductive reasoning tasks, presenting the premises in the same order as the ground truth proof in the prompt (as opposed to random ordering) drastically increases the model's accuracy. We first examine the effect of premise ordering on deductive reasoning on a variety of LLMs, and our evaluation shows that permuting the premise order can cause a performance drop of over 30%. In addition, we release the benchmark R-GSM, based on GSM8K, to examine the ordering effect for mathematical problem-solving, and we again observe a significant drop in accuracy, relative to the original GSM8K benchmark.

vladbogo abidlabs AshMariWalker HarshavardhanK clem Dlbk osanseviero ashwinlimaye marinaretik bunnycore alckasoc


SyntaxShap: Syntax-aware Explainability Method for Text Generation (2402.09259)

🔍 Today's pick in Interpretability & Analysis of LMs: SyntaxShap: Syntax-aware Explainability Method for Text Generation by @kamara000, R. Sevastjanova and M. El-Assady

Most model-agnostic post-hoc interpretability methods used nowadays in NLP were originally ported from tabular/CV domains with next to no adjustments to the intrinsic properties of textual inputs.

In this work, authors propose SyntaxSHAP, an adaptation of the Shapely value  approach in which coalitions used to compute marginal contributions to importance scores are constrained by the syntax of the explained sentence. The resulting tree-based coalitions do not satisfy the efficiency assumption of Shapley values but preserves the symmetry, nullity and additivity axioms.

SyntaxSHAP is compared to other model-agnostic approaches on small (GPT-2 117M) and large (Mistral 7B) LMs, showing it produces explanations that are more faithful to model predictions and more semantically meaningful than other common methods, while also being more efficient than the base SHAP method.

📄 Paper: https://huggingface.co/papers/2402.09259

💻 Code: https://github.com/k-amara/syntax-shap

🔍 All daily picks in LM interpretability: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9

lunarflu samusenps clem Dlbk RitaS sayhan


INSIDE: LLMs' Internal States Retain the Power of Hallucination
  Detection (2402.03744)

Eigenvalues to the rescue? 🛟🤔

I found out about this paper thanks to @gsarti 's post from last week; I got curious, so I want to post my take on it. 🤗

The paper proposes a new metric called EigenScore to detect LLM hallucinations. 📄

Their idea is that given an input question, they generate K different answers, take their internal embedding states, calculate a covariance matrix with them, and use it to calculate an EigenScore.

We can think of the EigenScore as the mean of the eigenvalues of the covariance matrix of the embedding space of the K-generated answers.

❓But why eigenvalues?

Well, if the K generations have similar semantics, the sentence embeddings will be highly correlated, and most eigenvalues will be close to 0.

On the other hand, if the LLM hallucinates, the K generations will have diverse semantics, and the eigenvalues will be significantly different from 0.

The idea is pretty neat and shows better results when compared to other methods like sequence probabilities, length-normalized entropy, and other uncertainty quantification-based methods.

💭 What I'm personally missing from the paper is that they don't compare their results with other methods like LLM-Eval and SelfcheckGPT. They do mention that EigenScore is much cheaper to implement than SelfcheckGPT, but that's all on the topic.

Paper: https://huggingface.co/papers/2402.03744

gsarti lunarflu EquinoxElahin samusenps clem zeusfsx linoyts Nikhil7280 Dlbk Parkerlambert123 dillfrescott Citaman shubhamg2208

osanseviero Dlbk gsarti lunarflu dillfrescott Citaman

AnaDP Esquare ashwinlimaye dillfrescott Pent anse10rville


Try out Mixtral 2-bit on a free-tier Google Colab notebook right now!

https://colab.research.google.com/drive/1-xZmBRXT5Fm3Ghn4Mwa2KRypORXb855X?usp=sharing

AQLM method has been recently introduced on transformers main branch

The 2bit model can be found here: https://huggingface.co/BlackSamorez/Mixtral-8x7b-AQLM-2Bit-1x16-hf-test-dispatch

And you can read more about the method here: https://huggingface.co/docs/transformers/main/en/quantization#aqlm

Great work @BlackSamorez and team!

osanseviero alielfilali01 sayhan Chunte victor samusenps macadeliccc clem Dlbk andysalerno jeffboudier loubnabnl linoyts medmac01 mathiasn1 Ji-Ha kramp bombassaro


The Aya project (https://huggingface.co/datasets/CohereForAI/aya_dataset, https://huggingface.co/datasets/CohereForAI/aya_collection and https://huggingface.co/datasets/CohereForAI/aya_evaluation_suite) by CohereForAI got released yesterday ! And today I'am excited to introduce Arabic Aya (2A) 🌟

Arabic Aya is a carefully curated dataset, derived from the vast Aya collection by CohereForAI, tailored specifically for Arabic language processing. It consolidates texts across Modern Standard Arabic (MSA) and other dialects, simplifying access to high-quality data for researchers, developers, and linguists.

🔍 Why Arabic Aya?
- Time-saving : Jump straight into your projects with pre-filtered Arabic texts.
- Diverse applications : Perfect for language modeling, sentiment analysis, dialect identification, and more.
- Community-driven : Your contributions and feedback can help enrich this resource further.

📚 Utilize Arabic Aya for your next NLP/LLM projects and be part of advancing Arabic language technologies. Let’s collaborate to make Arabic AI research more accessible and robust!

Check it out here: https://huggingface.co/datasets/2A2I/Arabic_Aya

vladbogo sayhan samusenps Dlbk ybelkada ajibawa-2023 osanseviero victor clem fqa-cyber kramp ArthurZ abdullah


Lumos : Empowering Multimodal LLMs with Scene Text Recognition (2402.08017)

Meta Reality Labs has developed Lumos, a system that merges Multimodal Large Language Models (MM-LLMs) with Scene Text Recognition (STR) to boost the efficiency of various tasks such as multimodal question-answering and text summarization.

Key aspects of Lumos include:

* Hybrid Computing: Utilizes a combination of on-device and cloud computing to process inputs, aiming to reduce latency.
* STR Components:
  * Region of Interest (ROI) Detection: Focuses on text-rich areas within images for optimized text extraction.
  * Text Detection and Recognition: Ensures high-quality text recognition within the ROI.
  * Reading Order Reconstruction: Arranges recognized text to mimic natural reading order, essential for context understanding.

Lumos demonstrates significant improvement with 80% accuracy in question-answering benchmarks and a low word error rate.

Paper: https://huggingface.co/papers/2402.08017

Congrats to the authors for their work!

Dlbk ybelkada ajibawa-2023 musfiqdehan osanseviero clem Esquare jjoker999 Mrdaffy bunnycore QuantumResearch merve ioanacroi


I think one of the most important ways you can contribute to open-source machine learning in 2024 is through datasets.

On Monday Argilla and Hugging Face launched #data-is-better-together an experiment focused on collectively building datasets on the Hub.

For our V1 experiment we're aiming to collectively rank 50k prompts!

 In the few days since launch we've had:

❤ 158 people contribute
🚀 2,796 prompts ranked

🤔 How Can You Contribute?

1. Sign up if you don’t have a Hugging Face account (why not!?)
2. Go to this Argilla Space and sign in: https://huggingface.co/spaces/DIBT/prompt-collective
3. Read the guidelines and start rating prompts!

You can also join the #data-is-better-together channel in the Hugging Face Discord 🔗 https://discord.com/channels/879548962464493619/1205128865735770142

yjernite samusenps vladbogo TuringsSolutions Dlbk ybelkada osanseviero santiviquez DmitryRyumin kramp julien-c clem dvilasuero pcuenq Sylvestre jeffboudier birgermoell Suparious mathiasn1 alvarobartt NovoCode matlok


NVIDIA released Chat with RTX. Why is it significant? It's NVIDIA's first step towards the vision of "LLM as Operating System" - a locally running, heavily optimized AI assistant that deeply integrates with your file systems, features retrieval as the first-class citizen, preserves privacy, and leverages your desktop gaming GPUs to the full.

A web-hosted version will have a hard time matching the same functionality. NVIDIA is going local even before OpenAI.

https://www.nvidia.com/en-us/ai-on-rtx/chat-with-rtx-generative-ai/

akhaliq victor osanseviero clem mrnemo97 PtiteTartine samusenps KADS-dev vladbogo Dlbk kramp J-Rojas mishig D4ve-R alielfilali01 cncqbaiying manoumhd99 abidlabs Jingya lysandre notune freddyaboulton

samusenps Dlbk mishig clem abidlabs Jingya lysandre notune


🚀Gradio Notebook (beta) is here! A custom component that deploys an interactive notebook UX on your Hugging Face space 🤗

➤ Deploy a Space in minutes without writing any UI code.
➤ Use a combination of text, image, and audio models in a single space
➤ Interact with models in a familiar notebook UI
➤ Shareable URL for model outputs to share with friends

👉 Check out the Space: https://huggingface.co/spaces/lastmileai/gradio-notebook-template
Demo video: https://youtu.be/FlwINB9RmKk
Repo: https://github.com/lastmile-ai/aiconfig/tree/main/gradio-notebook
Docs: https://aiconfig.lastmileai.dev/docs/gradio-notebook

Gradio Notebook was made possible thanks to Gradio 4.0, which introduced the ability for developers to create their own custom components and use them in Gradio apps.

Seeking feedback so please try it out and let us know how to improve the experience, both for space creators and consumers!

tanyarai akhaliq andrewlastmile rholinshead zakariassoul victor Rossdan multimodalart jonathanlastmileai julien-c clem samusenps vigsachi23 vladbogo sayhan ybelkada Dlbk bisnotforbella

akhaliq tanyarai Ankush-LM andrewlastmile rholinshead zakariassoul victor multimodalart julien-c clem vigsachi23 sayhan cnmoro ybelkada kramp Dlbk damerajee

akhaliq tanyarai andrewlastmile zakariassoul multimodalart julien-c clem vigsachi23 sayhan ybelkada Dlbk


BASE TTS: Lessons from building a billion-parameter Text-to-Speech model
  on 100K hours of data (2402.08093)

Amazon presents BASE TTS

Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data

https://huggingface.co/papers/2402.08093

introduce a text-to-speech (TTS) model called BASE TTS, which stands for Big Adaptive Streamable TTS with Emergent abilities. BASE TTS is the largest TTS model to-date, trained on 100K hours of public domain speech data, achieving a new state-of-the-art in speech naturalness. It deploys a 1-billion-parameter autoregressive Transformer that converts raw texts into discrete codes ("speechcodes") followed by a convolution-based decoder which converts these speechcodes into waveforms in an incremental, streamable manner. Further, our speechcodes are built using a novel speech tokenization technique that features speaker ID disentanglement and compression with byte-pair encoding. Echoing the widely-reported "emergent abilities" of large language models when trained on increasing volume of data, we show that BASE TTS variants built with 10K+ hours and 500M+ parameters begin to demonstrate natural prosody on textually complex sentences. We design and share a specialized dataset to measure these emergent abilities for text-to-speech. We showcase state-of-the-art naturalness of BASE TTS by evaluating against baselines that include publicly available large-scale text-to-speech systems: YourTTS, Bark and TortoiseTTS.

vladbogo Dlbk clem AshMariWalker alabulei AdinaY


There's a new leaderboard for vision language models 🤩
The models are ranked based on ELO, you can rate the responses to preselected examples or try with your input 🤗
https://huggingface.co/spaces/WildVision/vision-arena

kadirnar osanseviero YaTharThShaRma999 lunarflu Dlbk ashishkssingh clem samusenps sayhan vincentweisser maj34 sbrandeis


⚔ The TIGERLab's Text2Image arena is here! ⚔
 https://huggingface.co/spaces/TIGER-Lab/GenAI-Arena

Like https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard for LLMs: you prompt, two images emerge, vote for the best one  🏆

With enough votes this will lead to an Elo-based leaderboard for text-to-image models, go vote! 🗳
https://huggingface.co/spaces/TIGER-Lab/GenAI-Arena

osanseviero lunarflu vinesmsuic multimodalart seyf1elislam ldaniel-jmz OmbelineM


🚀 Exciting breakthrough in LLM reliability! 🧠NexusRaven-V2, our cutting-edge function-calling LLM, has set a new standard in minimizing AI hallucinations, surpassing GPT-4's performance in a recent third-party independent research benchmark.

Dive into our latest blog post to explore how we're pioneering reliable agents with minimal hallucinations: https://nexusflow.ai/blogs/towards-reliable-agents-with-minimal-hallucination

Key Highlights:

🏆 Zero Hallucinations: NexusRaven-V2 showcased remarkable accuracy with zero hallucinations in 840 tests, focusing on tool selection and usage – a significant leap over GPT-4 with 23 hallucinations.

📈 Enhanced Success Rates: It boasts a 9% higher success rate than GPT-4 in information-seeking applications requiring meticulous attention to detail and a 4% increase in adversarial scenarios that demand a deep understanding of tool documentation, even with vague tool and API argument names.

Try NexusRaven-V2 on Huggingface: https://huggingface.co/Nexusflow/NexusRaven-V2-13B

Check out the original third-party benchmark: https://arxiv.org/abs/2401.08326

osanseviero venkat-srinivasan-nexusflow brian-yu-nexusflow samusenps victor lvwerra Dlbk bunnycore fffiloni lewtun not-lain

osanseviero macadeliccc venkat-srinivasan-nexusflow brian-yu-nexusflow francis123 victor lvwerra Dlbk marinaretik jian-zhang-nexusflow

samusenps francis123 osanseviero lvwerra Dlbk tomaarsen not-lain tuantm jian-zhang-nexusflow sbrandeis

Dlbk ydeng9 not-lain jian-zhang-nexusflow


OS-Copilot: Towards Generalist Computer Agents with Self-Improvement (2402.07456)

OS-Copilot is a new framework for creating computer agents such as FRIDAY. This framework enables agents to interact seamlessly with your operating system, handling tasks like file management, multimedia editing, and more.

The system has three components:
* Planner: It takes complex user requests and breaks them down into manageable subtasks for efficient execution.
* Configurator: It prepares tasks for execution based on your preferences and available commands using a memory mechanism.
* Actor: It executes the tasks and learns from feedback, ensuring continuous improvement.

FRIDAY outperforms other methods on GAIA, a comprehensive benchmark. To answer the questions from GAIA, the agents need skills to calculate numbers, browse the web, process video and speech signal and others.

Resources:
* Paper: https://huggingface.co/papers/2402.07456
* Project GitHub: https://github.com/OS-Copilot/FRIDAY
* Project page: https://os-copilot.github.io/

Congrats to the authors Wu, Zhiyong et al. for their work!

clem macadeliccc Yamazaki-Hakubi victor sayhan dbalda Dlbk marinaretik Moibe


Fine-tune 7B models on free-tier Colab hardware using Unsloth 🦥

Unsloth is a framework for fine tuning language models boasting a 0% loss in accuracy  while using no approximation methods. They offer  a trainer for both supervised fine-tuning (SFT) and direct preference optimization (DPO) that can increase speed of fine-tuning by up to 5x.
This is achieved by adding LoRa adapters. This way they only need to train 1 to 10% of the total parameters. You can export a LoRa adapter or merge to 16-bit for a full finetune. The resulting model is prepared for use in vLLM for faster inference.

Additionally, Huggingface has integrated Unsloth into the documentation for DPO training and reported 18.6% performance gains on T4.

This sets a new standard for fine-tuning large language models. If you would like to explore this methodology for yourself I have provided a notebook "AutoSloth," where you can fine tune using either SFT or DPO and it will upload to HF with a prefilled Unsloth README 🦥 and a Q8_0 quantization.

The SFT example is set up for free tier usage, but the DPO example is set up for an A100. The DPO example can be altered to work on T4 but I wanted to include more than one example.

Colab Stats during training:
+ Model: unsloth/mistral-7b-bnb-4bit
+ Dataset: yahma/alpaca-cleaned
+ Batch size: 2
+ Gradient steps: 4
+ System RAM: 8.5 / 51.0 GB
+ VRAM (T4): 13.6 / 15.0 GB

Resources:
🦥Unsloth: https://github.com/unslothai/unsloth
🦥AutoSloth: https://colab.research.google.com/drive/1Zo0sVEb2lqdsUm9dy2PTzGySxdF9CNkc?usp=sharing
🤗HF-Unsloth-docs: https://huggingface.co/docs/trl/main/en/dpo_trainer#accelerate-dpo-fine-tuning-using-unsloth
🤗HF-Unsloth Blog Post: https://huggingface.co/blog/unsloth-trl

alielfilali01 clem sinanyuksel eramax samusenps osanseviero victor vladbogo ybelkada alvarobartt koalazf99 tomaarsen Citaman Dlbk davanstrien kolaposki julien-c Ji-Ha maywell Lovnir Krasi pandora-s boapps felicityworld pcuenq horiguchidotconf

ybelkada Dlbk julien-c felicityworld horiguchidotconf


@clem ist das der erste nicht Englische post auf huggingface?👋🏽 🇩🇪🇫🇷🇮🇹🇪🇸🇮🇳…

julien-c osanseviero Goofy-24 clem Fredithefish


Buffer Overflow in Mixture of Experts (2402.05526)

Mixture of experts: beware 🛡⚔

New paper by DeepMind: Buffer Overflow in MoE https://huggingface.co/papers/2402.05526

The paper shows an adversarial attack strategy in which a user sends malicious queries that can affect the output of other user queries from the same batch.

So if in the same batch we have
- User A benign query
- User B malicious query
The response for A might be altered!😱

How is this possible?
One approach is to fill the token buffers with adversarial data, hence forcing the gating to use the non-ideal experts or to entirely drop the bening tokens (in the case of finite limit size).

This assumes that the adversary can use the model as a black-box but can observe the logit outputs + ensure that the data is always grouped in the same batch.

How to mitigate this?
- Randomize batch order (and even run twice if some queries are very sensitive)
- Use a large capacity slack
- Sample from gate weights instead of top-k (not great IMO, as that require more memory for inference)

Very cool paper!!

NickyNicky julien-c macadeliccc ivanfioravanti clem samusenps victor JosephGesnouin pratjz blancsw marinaretik juancopi81 Dlbk djmcflush EmelMk surfiend Yasbok 0xe69756

macadeliccc clem samusenps JosephGesnouin blancsw santiviquez Dlbk abunchofrandomwords 0xe69756

meefs blancsw davanstrien Dlbk Yhyu13 Epiculous


SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step
  Reasoning (2308.00436)

Hey GPT, check yourself...

Here is a black-box method for hallucination detection that shows strong correlation with human annotations. 🔥

💡 The idea is the following: ask GPT, or any other powerful LLM, to sample multiple answers for the same prompt, and then ask it if these answers align with the statements in the original output. Make it say yes/no and measure the frequency with which the generated samples support the original statements.

This method is called SelfCheckGPT with Prompt and shows very nice results. 👀

The downside, we have to do many LLM calls just to evaluate a single generated paragraph... 🙃

More details and variations of this method are in the paper: https://huggingface.co/papers/2308.00436

arora-anmol osanseviero clem samusenps victor pulkitmehtawork Gvangelista yjernite


OS-Copilot

Towards Generalist Computer Agents with Self-Improvement

https://huggingface.co/papers/2402.07456

Autonomous interaction with the computer has been a longstanding challenge with great potential, and the recent proliferation of large language models (LLMs) has markedly accelerated progress in building digital agents. However, most of these agents are designed to interact with a narrow domain, such as a specific software or website. This narrow focus constrains their applicability for general computer tasks. To this end, we introduce OS-Copilot, a framework to build generalist agents capable of interfacing with comprehensive elements in an operating system (OS), including the web, code terminals, files, multimedia, and various third-party applications. We use OS-Copilot to create FRIDAY, a self-improving embodied agent for automating general computer tasks. On GAIA, a general AI assistants benchmark, FRIDAY outperforms previous methods by 35%, showcasing strong generalization to unseen applications via accumulated skills from previous tasks. We also present numerical and quantitative evidence that FRIDAY learns to control and self-improve on Excel and Powerpoint with minimal supervision. Our OS-Copilot framework and empirical findings provide infrastructure and insights for future research toward more capable and general-purpose computer agents.

morgan osanseviero clem samusenps angkul07 mindrage truong-xuan-linh impactframes


Fine-tuning LLMs is rad, but how do you manage all your checkpoints and evals in a production setting?

We partnered with @hamel to ship an Enterprise Model Management course packed full of learnings for those training, evaluating and deploying models at work.

Topics include:
- What webhooks are & how to use them to create integrations with different tools
- How to automate train -> eval runs
- Improving model governance and documentation
- Comparing candidate and baseline models
- Design patterns & recipes
- Lots more...

Would love to hear what you think!

👉  https://www.wandb.courses/courses/enterprise-model-management

osanseviero altryne macadeliccc clem samusenps


Show Me How It's Done: The Role of Explanations in Fine-Tuning Language
  Models (2402.07543)

🔍 Today's pick in Interpretability & Analysis of LMs: Show Me How It's Done: The Role of Explanations in Fine-Tuning Language Models by M. Ballout @krumnack et al.

Authors propose a fine-tuning procedure in which a classification task is framed as generation and augmented with a natural language explanation to clarify intermediate reasoning steps. The procedure is applied to fine-tune language models of various sizes on the ListOps dataset, containing synthetically-generated instructions on sequences of numbers.

Authors find that explanations contribute to improving model performances across all tested model sizes and explanations lengths. Smaller language models appear to benefit the most from this approach in terms of convergence speed, performance and input length generalisation, especially when given more exhaustive explanations.

📄 Paper: https://huggingface.co/papers/2402.07543

💻 Code: https://github.com/BalloutAI/Fine-tuning-with-Explanation

🔍 All daily picks in LM interpretability: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9


I love the new `Viewer` and i didn't knew how much i needed it until now
@sylvain , @lhoestq  and team, GREAT JOB 🔥 and THANK YOU 🤗

davanstrien osanseviero julien-c macadeliccc clem DamarJati kramp


In Search of the Robust Facial Expressions Recognition Model: The Visual Cross-Corpus Study,  @ElenaRyumina  et al.

Model: https://huggingface.co/ElenaRyumina/face_emotion_recognition
Demo: https://huggingface.co/spaces/ElenaRyumina/Facial_Expression_Recognition
Repo: https://github.com/ElenaRyumina/EMO-AffectNetModel

Paper: https://www.sciencedirect.com/science/article/pii/S0925231222012656

@ElenaRyumina

osanseviero clem samusenps victor fffiloni gryhkn kramp


🙋🏻♂hey there folks ,

🤗Aya has been released !  It's an absolutely massive undertaking to create a huge multilingual dataset and multilingual model of very high quality.

Papers :
https://cohere.com/research/papers/aya-dataset-paper-2024-02-13
https://cohere.com/research/papers/aya-model-paper-2024-02-13

Model : https://huggingface.co/CohereForAI/aya-101
Dataset : https://huggingface.co/datasets/CohereForAI/aya_dataset

I am proud to be one of 3,000 humans who built Aya - a new massively multilingual, generative LLM that outperforms existing open-source models and covers 101 different languages. Together, we are accelerating multilingual AI. 🤗

tellarin mvonwyl marcvw-sightengine osanseviero davanstrien kramp jnemecek clem samusenps raghavprabhakar MexIvanov sarahooker alielfilali01 yjernite


Aya Dataset: An Open-Access Collection for Multilingual Instruction
  Tuning (2402.06619)

Aya Dataset

An Open-Access Collection for Multilingual Instruction Tuning

https://huggingface.co/papers/2402.06619

Datasets are foundational to many breakthroughs in modern artificial intelligence. Many recent achievements in the space of natural language processing (NLP) can be attributed to the finetuning of pre-trained models on a diverse set of tasks that enables a large language model (LLM) to respond to instructions. Instruction fine-tuning (IFT) requires specifically constructed and annotated datasets. However, existing datasets are almost all in the English language. In this work, our primary goal is to bridge the language gap by building a human-curated instruction-following dataset spanning 65 languages. We worked with fluent speakers of languages from around the world to collect natural instances of instructions and completions. Furthermore, we create the most extensive multilingual collection to date, comprising 513 million instances through templating and translating existing datasets across 114 languages. In total, we contribute four key resources: we develop and open-source the Aya Annotation Platform, the Aya Dataset, the Aya Collection, and the Aya Evaluation Suite. The Aya initiative also serves as a valuable case study in participatory research, involving collaborators from 119 countries. We see this as a valuable framework for future research collaborations that aim to bridge gaps in resources.

NeuralNovel alielfilali01 Pclanglais taufiqdp kargaranamir samusenps Dlbk macadeliccc osanseviero vumichien juancopi81 lhoestq davanstrien julien-c mathiasn1 clem raghavprabhakar sugatoray DataSoul Ryukijano grenwi

alielfilali01 osanseviero vumichien alvarobartt davanstrien julien-c clem Dlbk Ryukijano


Today I'm releasing marginalia, a python library to perform corpus analysis and retrieve structured annotations with open LLMs like Mistral Open-Hermes-2.5: https://github.com/Pleias/marginalia

marginalia leverages vllm inference speed to re-generate until all the output matches an expected json structure and to send batches of several unstructured elements for enhanced patterns detections. It works especially well for bibliographies. The demo transforms a very old list (Benjamin Franklin favorite's books from 1744) into well-structured data: https://colab.research.google.com/drive/1xKjK2mDDpXMaKG5YLpFhOM7jehxt0kEt?usp=sharing

While marginalia can be quite flexible, it definitely isn't a general purpose tool for json generation (like outlines). I don't intend so far to extend support to more complex json structure, but really looking forward potential feedbacks and suggestions.

davanstrien osanseviero merve AIIAR samusenps gate369 macadeliccc clem dillfrescott


Grandmaster-Level Chess Without Search (2402.04494)

Google released a paper on Chess that doesn't rely on MCTS (aka AlphaZero) ♟
their secret sauce is.. synthetic data pseudolabeled by Stockfish engine 😀
2024 really is the year of synthetic data across all domains!
There's a nice discussion here, join us https://huggingface.co/papers/2402.04494

alielfilali01 osanseviero AIIAR samusenps nickandbro Citaman clem cosmojg


🤗 Data is better together!

Data is essential for training good AI systems. We believe that the amazing community built around open machine learning can also work on developing amazing datasets together.

To explore how this can be done, Argilla and Hugging Face are thrilled to announce a collaborative project where we’re asking Hugging Face community members to build a dataset consisting of LLM prompts collectively.

What are we doing?
Using an instance of Argilla — a powerful open-source data collaboration tool — hosted on the Hugging Face Hub, we are collecting ratings of prompts based on their quality.

How Can You Contribute?
It’s super simple to start contributing:

1. Sign up if you don’t have a Hugging Face account

2. Go to this Argilla Space and sign in: https://huggingface.co/spaces/DIBT/prompt-collective

3. Read the guidelines and start rating prompts!

You can also join the #data-is-better-together channel in the Hugging Face Discord.

Finally, to track the community progress we'll be updating this Gradio dashboard:

https://huggingface.co/spaces/DIBT/prompt-collective-dashboard

davanstrien gabrielmbmb osanseviero ignacioct sdiazlor plaguss julien-c tomaarsen nickandbro victor lunarflu clem frascuchon abbanbhan lewtun merve samusenps MoritzLaurer johko lhoestq smangrul sayhan medmac01 maghwa

davanstrien gabrielmbmb osanseviero kramp ignacioct sdiazlor julien-c victor severo clem mvaloatto lewtun samusenps chkla smangrul Muttermal

davanstrien osanseviero sdiazlor julien-c victor clem smangrul


Created a collection with all the LLM hallucination and evaluation papers I've been reading over the last few weeks 📄📄

https://huggingface.co/collections/santiviquez/llm-hallucination-detection-papers-65c4d2399096960aa80776d3

Which one should I add to the list?

osanseviero clem gsarti merve samusenps alielfilali01


Model Editing with Canonical Examples (2402.06155)

🔍 Today's pick in Interpretability & Analysis of LMs: Model Editing with Canonical Examples by @johnhew @sachen @lora-x E. Adams P. Jiang @manning

This works introduces a model editing approach using individual “canonical” examples to showcase desired/unwanted behavior. An evaluation is then conducted on out-of-distribution samples spanning six datasets (3 introduced in this work) covering settings of interest in bias mitigation, hard syntactic constructions and knowledge-based predictions, while limiting the degradation of the original model’s loss.

Authors experiment with Pythia LMs, finding that LoRa fine-tuning on canonical examples outperforms other established editing methods such as MEMIT.

Then, the approach is tested on Backpack LMs, using a linear combination of sense vectors to disentangle semantic information in the input texts. In particular, authors introduce “sense fine-tuning” where only a handful of sense vectors is updated per example, which is shown to be more efficient yet more effective than regular fine-tuning.

Finally, the relation between the predictions of pre- and post-sense fine-tuning backpack LMs is used to successfully transfer the desired adaptation to a larger standard LM, at no performance cost.

📄 Paper: https://huggingface.co/papers/2402.06155

🔍 All daily picks in LM interpretability: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9


3 hours between the two pictures 🔥
Finally that paranoid inside me got some rest 😂

NeuralNovel Skier8402 Wauplin tomaarsen osanseviero clem merve macadeliccc 9voltfan2009


Hear, hear, AMD MI300Xs have started to emerge much sooner than expected.

Here is a 2-part benchmarks report on performing BLOOM-176B inference using @MSFTDeepSpeed optimized for AMD MI300X.

1. https://www.evp.cloud/post/diving-deeper-insights-from-our-llm-inference-testing
2. https://www.evp.cloud/post/diving-deeper-insights-from-our-llm-inference-testing-part-2

This was published in response to our BLOOM-176B super-fast inference blog post https://huggingface.co/blog/bloom-inference-pytorch-scripts

Note that these have 192GB of HBM!

The NVIDIA monopoly is strong, but it'll have to start sharing the pie and hopefully drive the costs down at least somewhat.

Thanks to https://www.linkedin.com/in/eliovp for sharing this writeup with me.

p.s. at the PyTorch conference in the fall, the AMD representative said we will see MI300X available to us mortals in Q4-2024/Q1-2025.

osanseviero julien-c davanstrien tomaarsen VictorSanh clem pcuenq muhtasham merve samusenps Eliovp andrewrreed rwightman clefourrier


Introducing Remove Background Web: In-browser background removal, powered by @briaai's new RMBG-v1.4 model and 🤗 Transformers.js!

Everything runs 100% locally, meaning none of your images are uploaded to a server! 🤯 At only ~45MB, the 8-bit quantized version of the model is perfect for in-browser usage (it even works on mobile).

Check it out! 👇
Demo: https://huggingface.co/spaces/Xenova/remove-background-web
Model: https://huggingface.co/briaai/RMBG-1.4

Felladrin osanseviero not-lain baotrinh samusenps mgfeller munirdin mvaloatto DamarJati AIIAR eskayML OriLib HDiffusion brianjking 3ginger ashishtanwer NeuralNovel taufiqdp kramp Waltert enzostvs victor merve nickandbro ecarbo raphaelmansuy Seuriin Renadi rreed-pha Noomam dillfrescott


Introducing UNA-SimpleSmaug-34b:

Based on Smaug-34B-v0.1, capable of slightly outperform his base model and with increased math and reasoning thanks to simple-math dataset.
The model exhibits a great performance across diverse tasks with an excellent and balanced behaviour.
It scores 77.41 AVG on the Leaderboard, landing on #1 Position of 34B models.

Available in the hub already:
https://huggingface.co/fblgit/UNA-SimpleSmaug-34b-v1beta
https://huggingface.co/datasets/fblgit/simple-math

In this case, we applied UNA to the Attention Layers of the model while performing SFT with simple-math on a high complexity generated data of mathematics, proving the effect of simple-math on LLM's.

fblgit samusenps macadeliccc osanseviero mrfakename Ji-Ha NeuralNovel taufiqdp merve linuxhackr


Repo with scripts to create your own moe models using Apple mlx is constantly updated by @mzbac here: https://github.com/mzbac/mlx-moe

It's an amazing resource to learn inner workings of lora on moe with mlx.
It uses https://huggingface.co/datasets/WizardLM/WizardLM_evol_instruct_70k as default dataset, but can be easily tweak to use any model or dataset fro HF

Have fun with it!

Pavarissy osanseviero awni NeuralNovel clem merve sugatoray


✨ In-context learning is all you need!

This super interesting paper shows that fine-tuning with #SFT or #RLHF only helps on the form but does not impact knowledge or reasoning abilities, and in some cases, actually decreases performance!

They tested it with Mistral-base vs Mistral FT-ed, as well as Llama 2 70b base and FT-ed and results are consistent.

Providing the right prompt to the base model actually makes the model better and has 0 training cost!

Paper: https://arxiv.org/abs/2312.01552

andysalerno samusenps osanseviero gl198976 VikramSingh178 akjindal53244 nickandbro clem victor anthonymikinka


🎉🥳🎉
Today, we are thrilled to officially launch the "2A2I" Arabic Artificial Intelligence Initiative. This is a community-driven initiative founded on the philosophy of "Small team, Big work" Our goal is to elevate Arabic AI (LLMs, Diffusion Models, ASR, etc.) to the same level as English (and also Chinese 🐉).

Naturally, our focus today is primarily on datasets. We aim to provide high-quality datasets, especially for LLMs this month, to support our future efforts. In line with this, we're excited to introduce the Arabic version of H4-no_robots, find here : https://huggingface.co/datasets/2A2I/H4_no_robots (and yes, we know it's not "no_robots" anymore 😄). Stay tuned for more exciting, high-quality datasets in the next couple of weeks (+ 4 million rows🔥)

In parallel, we're also developing a model 🐪 that we hope will set new high standards for Arabic LLMs. 🔥 This model is planned for release in the coming months.

For more information, please visit our Organization card here : https://huggingface.co/2A2I

If you're interested in Arabic AI and want to help pushing the wheel as well, fill out this form, and let us know your motivation and your exciting ideas 🔥

The form link : https://forms.gle/kZLVuynWFU2FyTm57

If you have any questions, feel free to reach out to us at the email address below.

Additionally, if you believe as we do in this mission and would like to help this community and contribute some compute resources 😉 or any other form of help you might think about, please contact us at the same email address below or reach out to me through LinkedIn 🔥

2A2I Contact Email : arabic.ai.initiative@gmail.com
My LinkedIn : https://www.linkedin.com/in/alielfilali01/

medmac01 samusenps macadeliccc HANTIFARAH andysalerno Rami osanseviero pain NeuralNovel kramp Kernel clem kenza-ily maghwa derek-thomas SaiedAlshahrani mahmoudtarek


Today I found out about the existence of https://huggingface.co/utter-project/EuroLLM-1.7B-Instruct and unexpectedly it is really good. I think it's a very underrated model - give it a try https://huggingface.co/spaces/nyuuzyou/EuroLLM-1.7B-Instruct


Ever wondered how neural networks actually work under the hood?

In my latest video, I break down the core mathematical concepts behind neural networks in a way that's easy for IT professionals to understand. We'll explore:

- Neurons as logic gates
- Weighted sums and activation functions
- Gradient descent and backpropagation

No complex equations or jargon, just clear explanations and helpful visuals!

➡ Watch now and unlock the mysteries of neural networks: https://youtu.be/L5_I1ZHoGnM


Philosopher Gilles Deleuze in 1985-86 about society of control, probabilities, and power. Visionary words in an era of autoregressive models:

"The biopolitics of populations appears when right sets about administering life, says Foucault, administering life in any open multiplicities whatever. You see the importance of the difference between discipline and biopolitics. The one is in an open space, with large multiplicities to which limits are not assignable. They can only be treated by the calculus of probabilities, hence the development of the calculus of probabilities and the meaning [sens] of the social control of probabilities, the probabilities of marriage in a nation, the probabilities of mortality, probabilities of natality. Natality, nuptiality, mortality …

... When Foucault directly addresses the question of power, namely, one of his great theses: no, power does not repress, or it represses only secondarily. What does it do? It does something much more profound and, doubtless, more formidable that repressing: it forms, it shapes. It does not silence, it does worse: it makes speak. It disciplines, it standardizes [normalise]. But repression is entirely secondary in relation to the positive operations of power.

Power does not repress, it disciplines, it manages, it controls, it standardizes, etcetera. It does not silence, it makes speak. It does not prevent acting, it makes act."

From the Deleuze Seminars at Université Paris 8 translated by Purdue University -> https://deleuze.cla.purdue.edu/


The Synthetic Data Generator now directly integrates with Argilla, so you can generate and curate your own high-quality datasets from pure natural language!

Up next -> include dataset generation for text classification.
Other suggestions? Let us know.

Space: https://huggingface.co/spaces/argilla/synthetic-data-generator


While Google's Transformer might have introduced "Attention is all you need," Microsoft and Tsinghua University are here with the DIFF Transformer, stating, "Sparse-Attention is all you need."

The DIFF Transformer outperforms traditional Transformers in scaling properties, requiring only about 65% of the model size or training tokens to achieve comparable performance.

The secret sauce? A differential attention mechanism that amplifies focus on relevant context while canceling out noise, leading to sparser and more effective attention patterns.

How?
- It uses two separate softmax attention maps and subtracts them.
- It employs a learnable scalar λ for balancing the attention maps.
- It implements GroupNorm for each attention head independently.
- It is compatible with FlashAttention for efficient computation.

What do you get?
- Superior long-context modeling (up to 64K tokens).
- Enhanced key information retrieval.
- Reduced hallucination in question-answering and summarization tasks.
- More robust in-context learning, less affected by prompt order.
- Mitigation of activation outliers, opening doors for efficient quantization.

Extensive experiments show DIFF Transformer's advantages across various tasks and model sizes, from 830M to 13.1B parameters.

This innovative architecture could be a game-changer for the next generation of LLMs. What are your thoughts on DIFF Transformer's potential impact?


MixGen3 is an innovative image generation service that utilizes LoRA (Low-Rank Adaptation) models. Its key features include:

Integration of various LoRA models: Users can explore and select multiple LoRA models through a gallery.
Combination of LoRA models: Up to three LoRA models can be combined to express unique styles and content.
User-friendly interface: An intuitive interface allows for easy model selection, prompt input, and image generation.
Advanced settings: Various options are provided, including image size adjustment, random seed, and advanced configurations.

Main applications of MixGen3:

Content creation
Design and illustration
Marketing and advertising
Education and learning

Value of MixGen3:

Enhancing creativity
Time-saving
Collaboration possibilities
Continuous development

Expected effects:

Increased content diversity
Lowered entry barrier for creation
Improved creativity
Enhanced productivity

MixGen3 is bringing a new wave to the field of image generation by leveraging the advantages of LoRA models. Users can experience the service for free at
https://openfree-mixgen3.hf.space

contacts: arxivgpt@gmail.com

aiqtech seawolf2357 fantos aiqcamp ginipick openfree DeathGodlike Sakalti Moibe

aiqtech seawolf2357 fantos aiqcamp ginipick openfree Ar4ikov

aiqtech seawolf2357 fantos fantaxy openfree John6666

aiqtech seawolf2357 aiqcamp ginipick fantaxy openfree

aiqtech seawolf2357 fantos ginipick openfree

aiqtech seawolf2357 fantos aiqcamp openfree


Model is always disabled?
#script...
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("distilbert/distilgpt2",
                                             token="xxxxxx")

That loads the model fine.  But if used by index returned from VectorStoreIndex for QDrant like this:

#script...
query_engine = index_from_nodes.as_query_engine(llm=model, streaming=True)

response = query_engine.query(
    "What is formula 1?"
)

response.print_response_stream()

It errors out with a disabled error:
AssertionError                            Traceback (most recent call last)
Cell In[34], line 1
----> 1 query_engine = index_from_nodes.as_query_engine(llm=model, streaming=True)
      3 response = query_engine.query(
      4     "What is formula 1?"
      5 )
      7 response.print_response_stream()

File ~/miniconda/lib/python3.9/site-packages/llama_index/core/indices/base.py:376, in BaseIndex.as_query_engine(self, llm, **kwargs)
    370 from llama_index.core.query_engine.retriever_query_engine import (
    371     RetrieverQueryEngine,
    372 )
    374 retriever = self.as_retriever(**kwargs)
    375 llm = (
--> 376     resolve_llm(llm, callback_manager=self._callback_manager)
    377     if llm
    378     else Settings.llm
    379 )
    381 return RetrieverQueryEngine.from_args(
    382     retriever,
    383     llm=llm,
    384     **kwargs,
    385 )

File ~/miniconda/lib/python3.9/site-packages/llama_index/core/llms/utils.py:102, in resolve_llm(llm, callback_manager)
     99     print("LLM is explicitly disabled. Using MockLLM.")
    100     llm = MockLLM()
--> 102 assert isinstance(llm, LLM)
    104 llm.callback_manager = callback_manager or Settings.callback_manager
    106 return llm

AssertionError:

So why is the LLM disabled?
Thanks!


📢 We are giving extra two weeks before switching to the final stage of RuOpinionNE-2024.
⏰  The final stage starts since 1-st of November 2024.
We have already first baseline submission by 👨💻 RefalMachine that showcase F1 = 0.17  based on Qwen2 model series.

For those who wish to attend:
📊 Codalab: https://codalab.lisn.upsaclay.fr/competitions/20244
🗒 Task: https://codalab.lisn.upsaclay.fr/competitions/20244#learn_the_details-overview
🔔 Updates: https://t.me/RuOpinionNE2024

🙋 Questions: https://nicolay-r.github.io/
🧪 Past experiments: https://github.com/nicolay-r/RuSentNE-LLM-Benchmark


Professors should ask students to write blog posts based on their final projects instead of having them do paper-like reports.

A single blog post, accessible to the entire internet, can have a greater career impact than dozens of reports that nobody will read.


Who's going to get to the most liked model on Hugging Face first: StabilityAI, Meta, Black Forest or someone else? The race is on!


The Truth is in There: Improving Reasoning in Language Models with
  Layer-Selective Rank Reduction (2312.13558)

Reducing perplexity in LLM's through layer selective rank reduction

Layer-Selective Rank Reduction (LASER) is a denoising method that improves reasoning by the strategic removal of higher-order components from weight matrices in the multi-layer perceptron (MLP) layers without the need for additional parameters or training data. This process leverages singular value decomposition to identify and eliminate these components. This simple, yet effective, method has shown to improve question-answering performance by up to 27.4 percentage points.

LaserRMT implements this through a process by calculating signal to noise ratio (SNR) for each layer and selectively reducing the rank of these layers.The SNR method meticulously computes the SNR by leveraging singular value decomposition (SVD) to separate the signal (higher-order components) from the noise (lower-order components) within the weight matrices of the model's layers. The SNR calculation is what determines which layers would benefit from rank reduction without compromising the models integrity.

If a layer is identified that could benefit from rank reduction, then the layer will enter an incremental process where the weight matrices are reduced and reconstructed by retaining only the singular values that surpass the threshold. In the case of laserRMT, the threshold is calculated by Marchenko-Pastur Law.
```
@staticmethod
    def marchenko_pastur_threshold(sigma, n, m):
        beta = n / m if n < m else m / n
        threshold = sigma * np.sqrt((1 + np.sqrt(beta))**2)
        return thr
```
The two primary benefits of applying this method are reducing computational overhead of large language models and simultaneously improving output quality.

Credit to @ehartford @fernandofernandes @DavidGF for laserRMT

Resources:
☄ AutoLaser: https://colab.research.google.com/drive/11j0e-w6BfvqeFN1gUrpOqdW0vcKqfVqP?usp=sharing
laserRMT: https://github.com/cognitivecomputations/laserRMT
https://huggingface.co/papers/2312.13558

samusenps DavidGF Undi95 mathiasn1 osanseviero KnutJaegersberg tiendung Ji-Ha Dlbk munirdin NeuralNovel victor Citaman abdullah


More Agents Is All You Need (2402.05120)

More Agents Is All You Need

https://huggingface.co/papers/2402.05120

find that, simply via a sampling-and-voting method, the performance of large language models (LLMs) scales with the number of agents instantiated. Also, this method is orthogonal to existing complicated methods to further enhance LLMs, while the degree of enhancement is correlated to the task difficulty. We conduct comprehensive experiments on a wide range of LLM benchmarks to verify the presence of our finding, and to study the properties that can facilitate its occurrence.


AttnLRP: Attention-Aware Layer-wise Relevance Propagation for
  Transformers (2402.05602)

🔍 Today's pick in Interpretability & Analysis of LMs: AttnLRP: Attention-Aware Layer-wise Relevance Propagation for Transformers by @RedOneAI et al.

This work proposes extending the LRP feature attribution framework to handling Transformers-specific layers. In particular: authors:

1. Propose a generalized approach to softmax linearization by designing a distribution rule that incorporates bias terms, absorbing of a portion of the relevance.
2. Propose decomposing the element-wise matrix multiplication in the attention operation as a sequential of epsilon and uniform distribution rules to ensure conservation (=sum of relevance stays constant across layers)
3. Propose handling normalisation layers with an identity distribution rule.

By means of extensive experiments, authors show that AttnLRP:

1. Is significantly more faithful than other popular gradient- and attention-based attribution approaches on CV and NLP tasks using large transformer models.
2. Runs in O(1) time, requiring O(sqrt(num_layers)) memory, as opposed to perturbation-based approaches requiring O(seq_len) time.
3. can be used alongside activation maximisation to explain the contribution of granular model components in driving models’ predictions.

📄 Paper: https://huggingface.co/papers/2402.05602

🔍 All daily picks in LM interpretability: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9

osanseviero RedOneAI erfanhatefi alielfilali01 OctoOptLab


We have recently shipped two new task pages thanks to the contributors!
Both of these pages consists of models you can use off-the-shelf 💜
Reading these you can start building in no time 🛠
Task page for zero-shot object detection 👉  https://huggingface.co/tasks/zero-shot-object-detection 📚
Task page for mask generation 👉  https://huggingface.co/tasks/mask-generation 📖

fffiloni kramp osanseviero macadeliccc AdinaY not-lain nbroad

jeffboudier santiviquez macadeliccc not-lain alielfilali01


GlotLID: Language Identification for Low-Resource Languages (2310.16248)

A Text Language Identification Model with Support for +2000 Labels:

space: https://huggingface.co/spaces/cis-lmu/glotlid-space
model: https://huggingface.co/cis-lmu/glotlid
github: https://github.com/cisnlp/GlotLID
paper: https://huggingface.co/papers/2310.16248

s3nh tomaarsen merve kardal Sylvestre kramp osanseviero macadeliccc kaizuberbuehler FPK39 lbourdois DmitryRyumin sbarman25 ayymen


The most widely used French NER models on HF (https://huggingface.co/Jean-Baptiste/camembert-ner and https://huggingface.co/cmarkea/distilcamembert-base-ner) are trained on a single dataset (WikiNER) which on the one hand contains leaks and therefore distorts the true results of these models, and on the other hand overspecializes them in a particular domain (= texts from Wikipedia). They are also only available in a base version (110M parameters).

That's why I've trained new NER models in French both on more data (x3), as well as in base and large versions (336M). They are available in 3 entities (PER, ORG, LOC) or 4 entities (PER, ORG, LOC, MISC):
- https://huggingface.co/CATIE-AQ/NERmembert-base-4entities
- https://huggingface.co/CATIE-AQ/NERmembert-large-4entities
- https://huggingface.co/CATIE-AQ/NERmembert-base-3entities
- https://huggingface.co/CATIE-AQ/NERmembert-large-3entities

Datasets without leaks are also available:
- https://huggingface.co/datasets/CATIE-AQ/frenchNER_4entities
- https://huggingface.co/datasets/CATIE-AQ/frenchNER_3entities

osanseviero Tonic EquinoxElahin mishig tomaarsen


🚀 It's now easier than ever to switch from OpenAI to open LLMs

Hugging Face's TGI now supports an OpenAI compatible Chat Completion API

This means you can transition code that uses OpenAI client libraries (or frameworks like LangChain 🦜  and LlamaIndex 🦙) to run open models by changing just two lines of code 🤗

⭐ Here's how:
```
from openai import OpenAI

# initialize the client but point it to TGI
client = OpenAI(
    base_url="<ENDPOINT_URL>" + "/v1/",  # replace with your endpoint url
    api_key="<HF_API_TOKEN>",  # replace with your token
)
chat_completion = client.chat.completions.create(
    model="tgi",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Why is open-source software important?"},
    ],
    stream=True,
    max_tokens=500
)

# iterate and print stream
for message in chat_completion:
    print(message.choices[0].delta.content, end="")
```

🔗 Blog post ➡ https://huggingface.co/blog/tgi-messages-api
🔗 TGI docs ➡ https://huggingface.co/docs/text-generation-inference/en/messages_api

osanseviero clem samusenps ashesashes johko Tonic julien-c victor Chunte johnnybio Alisherfullhd Yogeek Glavin001 nbroad Nymbo

samusenps hunkim kramp Tonic julien-c Chunte jeffboudier natolambert Felladrin Nymbo

mvaloatto hunkim Tonic julien-c Glavin001 Nymbo


First HF social post:

pip install -U mlx

osanseviero clem ivanfioravanti multimodalart johko merve victor reach-vb hfthorvaldur flymonk

osanseviero clem multimodalart merve kardal reach-vb


Apple MLX 0.2.0 is here!

@awni and his team did again! Mega release.

Performance 💨💨💨  is coming to town!
mx.compile makes stuff go fast
- Some functions are up to 10x faster (benchmarks: https://github.com/ml-explore/mlx/pull/614#issuecomment-1929056286)
- Training models anywhere from 10% to twice as fast (benchmarks: https://github.com/ml-explore/mlx-examples/pull/420#issuecomment-1932422605)

Full details on this release here: https://github.com/ml-explore/mlx/releases/tag/v0.2.0

awni osanseviero clem heyyeah samusenps akashicmarga merve victor kroonen Gatozu35


Benefits of `imatrix` quantization in place of quip#

Quip-# is a quantization method proposed by [Cornell-RelaxML](https://github.com/Cornell-RelaxML) that claims tremendous performance gains using only 2-bit precision.

RelaxML proposes that quantizing a model from 16 bit to 2 bit precision they can utilize Llama-2-70B on a single 24GB GPU.

QuIP# aims to revolutionize model quantization through a blend of incoherence processing and advanced lattice codebooks. By switching to a Hadamard transform-based incoherence approach, QuIP# enhances GPU efficiency, making weight matrices more Gaussian-like and ideal for quantization with its improved lattice codebooks.

This new method has already seen some adoption by projects like llama.cpp. The use of the Quip-# methodology has been implemented in the form of imatrix calculations. The importance matrix is calculated from a dataset such as wiki.train.raw and will output the perplexity on the given dataset.

This interim step can improve the results of the quantized model. If you would like to explore this process for yourself:

llama.cpp - https://github.com/ggerganov/llama.cpp/
Quip# paper - https://cornell-relaxml.github.io/quip-sharp/
AutoQuip# colab -  https://colab.research.google.com/drive/1rPDvcticCekw8VPNjDbh_UcivVBzgwEW?usp=sharing

Other impressive quantization projects to watch:
+ AQLM
https://github.com/Vahe1994/AQLM
https://arxiv.org/abs/2401.06118

nivu clem merve osanseviero ybelkada samusenps NeuralNovel Ji-Ha hllj DamarJati sayhan NovoCode sugatoray 1TuanPham victor John6666 louisbrulenaudet

clem merve osanseviero ybelkada NeuralNovel Gatozu35 akjindal53244 sayhan NovoCode KingNish


Corrective Retrieval Augmented Generation (2401.15884)

What if the retrieval goes wrong? 🐕

Retrieval Augmented Generation (RAG) is a strategy to alleviate LLM hallucinations and improve the quality of generated responses.

A standard RAG architecture has two main blocks: a Retriever and a Generator.

1⃣ When the system receives an input sequence, it uses the Retriever to retrieve the top-K most relevant documents associated with the input sequence. These documents typically come from an external source (e.g., Wikipedia) and are then concatenated to the original input's context.

2⃣ It then uses the Generator to generate a response given the gathered information in the first step.

But what happens if the retrieval goes wrong and the retrieved documents are of very low quality?

Well, in such cases, the generated response will probably be of low quality, too. 🫠

But here is where CRAG (Corrective RAG) *might* help. I say it might help because the paper is very new — only one week old, and I don't know if someone has actually tried this in practice 😅

However, the idea is to add a Knowledge Correction block between the Retrieval and Generation steps to evaluate the retrieved documents and correct them if necessary.

This step goes as follows:

🟢 If the documents are correct, they will be refined into more precise knowledge strips and concatenated to the original context to generate a response.

🔴 If the documents are incorrect, they will be discarded, and instead, the system searches the web for complementary knowledge. This external knowledge is then concatenated to the original context to generate a response.

🟡 If the documents are ambiguous, a combination of the previous two resolutions is triggered.

The experimental results from the paper show how the CRAG strategy outperforms traditional RAG approaches in both short and long-form text generation tasks.

Paper: https://huggingface.co/papers/2401.15884

osanseviero samusenps NeuralNovel nickandbro abellion


Grandmaster-Level Chess Without Search

https://huggingface.co/papers/2402.04494

largest model reaches a Lichess blitz Elo of 2895 against humans, and successfully solves a series of challenging chess puzzles, without any domain-specific tweaks or explicit search algorithms. We also show that our model outperforms AlphaZero's policy and value networks (without MCTS) and GPT-3.5-turbo-instruct. A systematic investigation of model and dataset size shows that strong chess performance only arises at sufficient scale. To validate our results, we perform an extensive series of ablations of design choices and hyperparameters.

samusenps andysalerno osanseviero NeuralNovel


Prompts are hyperparameters. Every time you test a different prompt on your data, you become less sure if the LLM actually generalizes to unseen data.

Issues of overfitting to a test set seem like concepts from boring times when people still fine-tuned models, but it's just as important for "zeroshot prompting". Using a separate validation split to tune the main hyperparameter of LLMs (the prompt) is just as important as train-val-test splitting for fine-tuning. The only difference is that you don't have a training dataset anymore and it somehow feels different because there is no training / no parameter updates.

Its easy to trick yourself into believing that an LLM performs well on your task, while you've actually overfit the prompt on your data. Every good "zeroshot" paper should clarify that they used a validation split for finding their prompt before final testing.

samusenps tollefj merve osanseviero sayhan


Hi friends, i'am happy to share with you all a tool that i built a week ago or so, i'am talking here about the "LLM Training Cost Calculator" - a handy tool now available on Hugging Face Spaces! This interactive Gradio app provides an easy-to-use interface for estimating the training costs of large language models (LLMs).

(I've been asked to provide a report about the cost of finetuning each model etc... so i decided to do the lazy job and build a tool for it, Prof later can choose whatever config he likes 😆)

🔍 But Why this is important?
As LLMs continue to grow in size and complexity, understanding the computational and financial requirements is crucial for planning and managing AI projects. I believe this tool simplifies this process, giving you insights into potential expenses based on the number of parameters and tokens in your dataset.

🌟 Features:
- Input the number of parameters (in billions) and tokens (in trillions).
- Adjust for GPU utilization rates and overhead costs.
- Get an instant estimate of your training costs.
+ Choose your GPU (A100 80GB PCle, A100 80GB SXM, V100, H100 SXM, H100 PCle)

📈 Coming Soon:
Plans are in place to expand the calculator's capabilities to include fine-tuning costs for models using LoRA or QLoRA. You'll be able to input a model ID from the Hugging Face Hub, select your fine-tuning strategy, and specify quantization details if using QLoRA.

I believe this tool will be a valuable asset to the AI community, helping to plan and allocate resources more effectively 🤗.

Should you have any suggestions or feedback, please don't hesitate to contribute your thoughts in the comments below. Together, we can refine and enhance this resource for all.

🔗 Try it here : https://huggingface.co/spaces/Ali-C137/LLM-Training-Cost-Calculator

PS : All thanks to Gradio, Hugging Face and the community ofc 🔥 😉

victor merve Sylvestre osanseviero ivanfioravanti


A while ago, I presented this Phi2 DPO fine-tune notebook with LoRa. Got some input from @ybelkada  about not needing a `ref_model` because we can just swap out the LoRa adapters during training. Cool feature 🤓

https://colab.research.google.com/drive/1PGMj7jlkJaCiSNNihA2NtpILsRgkRXrJ#scrollTo=wXqoH2TMnjjp

merve samusenps ybelkada alielfilali01 sumandas compressionsavant wise-east


Welcome Bunny! A family of lightweight but powerful multimodal models from BAAI

With detailed work on dataset curation, the Bunny-3B model built upon SigLIP and Phi-2 achieves performance on par with 13B models.

Model: https://huggingface.co/BAAI/bunny-phi-2-siglip-lora


There appears to be a huge misunderstanding regarding the licensing requirements for open sourced Chinese speaking speaking LLMs on
@huggingface

I initially shared this misconception too, but after conducting some research, I came up with the list below.

Veryimpressive!

osanseviero samusenps AdinaY Tonic yahma Tom9000 wwizo aloobun sayhan


🌠 Let's try to figure out this one as a community.
What reactions should we add to Posts and discussions?

If the reaction you want is already in the replies give it a thumbs up (👍) if it's not just add it as a reply.

julien-c Sylvestre santiviquez kramp taufiqdp osanseviero ArthurZ samusenps clem mvaloatto Chunte


Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations
  from Large Language Models (2402.04614)

🔍 Today's pick in Interpretability & Analysis of LMs: Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Models by C. Agarwal, S.H. Tanneru and H. Lakkaraju

This work discusses the dichotomy between faithfulness and plausibility in LLMs’ self-explanations (SEs) in natural language (CoT, counterfactual reasoning, and token importance). These explanations tend to be reasonable according to human understanding (plausible) but are not always aligned with the reasoning processes of the LLMs (unfaithful).

Authors remark that the increase in plausibility driven by the request for a friendly conversational interface might come at the expense of faithfulness. Provided the faithfulness requirements of many high-stakes real-world settings, authors suggest these are considered when designing and evaluating new explanation methodologies.

Finally, the authors call for a community effort to 1) develop reliable metrics to characterize the faithfulness of explanations and 2) pioneering novel strategies to generate more faithful SEs.

📄 Paper: https://huggingface.co/papers/2402.04614

🔍 All daily picks in LM interpretability: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9


#lg #gram #solarllm BREAKING NEWS:

upstage/Solar LLM will soon be available for LG gram Laptops as an on-device LLM. 💻🌞🎉

Upstage makes LLMs accessible to everyone and every device. We'd love to see more on-device LLMs.

https://koreajoongangdaily.joins.com/news/2024-02-06/business/industry/LG-Electronics-signs-partnership-with-generative-AI-startup-Upstage-/1975528

samusenps leonardlin osanseviero hf-delta akashicmarga


From @IkariDev and @Undi95
New release from NeverSleep!

MiquMaid-v2-70B
https://huggingface.co/NeverSleep/MiquMaid-v2-70B
https://huggingface.co/NeverSleep/MiquMaid-v2-70B-GGUF
DPO version
https://huggingface.co/NeverSleep/MiquMaid-v2-70B-DPO
https://huggingface.co/NeverSleep/MiquMaid-v2-70B-DPO-GGUF

MiquMaid-v2-2x70B
https://huggingface.co/NeverSleep/MiquMaid-v2-2x70B
https://huggingface.co/NeverSleep/MiquMaid-v2-2x70B-GGUF
DPO version
https://huggingface.co/NeverSleep/MiquMaid-v2-2x70B-DPO
https://huggingface.co/NeverSleep/MiquMaid-v2-2x70B-DPO-GGUF

Feedback appreciated!

uncensorie frammie hunkim samusenps s3nh Waltert not-lain rewij okeksama IHaBiS


Are you up to a 🤗 challenge 🏆 ?
if so 👀 Check out the new MindBigData Leaderboard 🔥🔥🔥
🚀 https://huggingface.co/spaces/DavidVivancos/MindBigData-Leaderboard

Decode the "source" 🧠 with the largest multimodal opendata of brain signals for Machine Learning.

Try to beat the whopping 🥇98,97% accuracy of Smita Tiwari Shivani Goel School of CSET Bennett University, India and Arpit Bhardwaj BML Munjal University decoding the multiclass Yann LeCun mnist of brain digits caputured with EMOTIV Epoc and 🥇89,62% with Insight

Or the 🥇96,18% of Dr. Nrushingh Charan Mahapatra Intel Corporation and Prof.(Dr). Prachet Bhuyan Kalinga Institute of Industrial Technology, Bhubaneswar also with the mnist of brain digits but captured with Muse® by Interaxon Inc.

Or the 🥇85% of Matthew Zhang Westlake High School and Jeremy Lu now at Purdue University decoding brain images captured from imagenet

Or be the first to break the 🧊 with the largest open dataset 2023 (8+ billion datapoints), the multimodal MindBigData2023_MNIST-8B captured with a custom 128 channels EEG that I built and with the real 70,000 MNIST digits and put your NVIDIA gpus to work.

All the datasets are open and ready at HuggingFace, dare to try?

Hope to see you all soon in the LeaderBoard

Thanks
@DavidVivancos

TF1920 osanseviero clem samusenps victor julien-c clefourrier Tonic


EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameters (2402.04252)

EVA-CLIP 🦖 is the CLIP scaled to the moon! 🔥
The new SotA CLIP-like model 🏆
Highlights ✨
- Performs better in linear probing
- Outperforms in Zero-Shot Image-Text Retrieval
- Higher zero-shot accuracy in IN-1K

As usual, try it with the notebook I built for you https://colab.research.google.com/drive/1K7DdCORC3x4qyhwhuB4fT4wcfJ_BQLKw?usp=sharing#scrollTo=0ZS_lJ7SK6Ys
I also built a Space for you to compare the output probabilities to CLIP, seems that EVACLIP is more "sure" of it's results 😊 https://huggingface.co/spaces/merve/EVACLIP
The authors have shared 8B checkpoints open with Apache 2.0 license 💜 and it's built on top of transformers, super easy to use! https://huggingface.co/BAAI/EVA-CLIP-8B
Read the paper https://huggingface.co/papers/2402.04252 📄

victor osanseviero santiviquez clem AIIAR samusenps danielus boapps qiying


Super excited to share my project, ageML, here! 😊

ageML is a Python library I've been building to study the temporal performance degradation of ML models.

The goal of the project is to facilitate the exploration of performance degradation by providing tools for people to easily test how their models would evolve over time when trained and evaluated on different subsets of their data.

⭐ Check it out: https://github.com/santiviquez/ageml

victor osanseviero kramp clem hunkim samusenps merve abhishekvarma12345 MarinaraSpaghetti eevee32x


Self-Discover: Large Language Models Self-Compose Reasoning Structures (2402.03620)

Self-Discover

Large Language Models Self-Compose Reasoning Structures

paper page: https://huggingface.co/papers/2402.03620

SELF-DISCOVER substantially improves GPT-4 and PaLM 2's performance on challenging reasoning benchmarks such as BigBench-Hard, grounded agent reasoning, and MATH, by as much as 32% compared to Chain of Thought (CoT). Furthermore, SELF-DISCOVER outperforms inference-intensive methods such as CoT-Self-Consistency by more than 20%, while requiring 10-40x fewer inference compute. Finally, we show that the self-discovered reasoning structures are universally applicable across model families: from PaLM 2-L to GPT-4, and from GPT-4 to Llama2, and share commonalities with human reasoning patterns.

osanseviero Erland clem vaibhavgeek merve akashicmarga


Memphis: Advancing language model reasoning without relying on proprietary model outputs

Memphis is a series of models which advance human-data models, offering good performance without relying on proprietary model outputs (e.g. GPT-generated datasets). I've developed a new iterative finetuning procedure to improve the reasoning ability of these models beyond what is possible using only SFT on the same data.

Currently, I've released two models: Memphis-CoT-3B, and Memphis-scribe-3B.

To create these models, I've created new datasets:
- https://huggingface.co/datasets/euclaise/reddit-instruct : A dataset of instruction/QA-like data scraped from Reddit. A curated version, filtered using Lilac and neural embedding models, is available at https://huggingface.co/datasets/euclaise/reddit-instruct-curated
- https://huggingface.co/datasets/euclaise/TinyCoT : TinyCoT is a mtea-dataset that aggregates a variety of different human-sourced reasoning data. It is a curated version of my previous MegaCoT dataset https://huggingface.co/datasets/euclaise/MegaCoT, which contains 629k responses which get cut down to 28k for TinyCoT.  There's also an intermediate version https://huggingface.co/datasets/euclaise/MiniCoT, which has 129k responses.

Memphis-CoT is trained on reddit-instruct, a filtered version of oasst2 https://huggingface.co/datasets/sablo/oasst2_curated, and TinyCoT. Multiple iterations were performed on TinyCoT, while reddit-instruct and oasst2 were only used for the initial model.

Memphis-scribe further finetunes Memphis-CoT on more creative tasks. It was finetuned from Memphis-CoT on 18 different datasets, including datasets like https://huggingface.co/datasets/euclaise/WritingPrompts_curated, https://huggingface.co/datasets/lemonilia/LimaRP, and more.

To prevent catastrophic forgetting, I used weight averaging between iterations.

- https://huggingface.co/euclaise/Memphis-CoT-3B
- https://huggingface.co/euclaise/Memphis-scribe-3B

osanseviero lbourdois victor samusenps xsa-dev merve ABIN1012 d0rj alielfilali01 reciprocate HailJebus qqqzzzyyy