AtomoVideo: High Fidelity Image-to-Video Generation (2403.01800)

AtomoVideo

High Fidelity Image-to-Video Generation

https://huggingface.co/papers/2403.01800

Recently, video generation has achieved significant rapid development based on superior text-to-image generation techniques. In this work, we propose a high fidelity framework for image-to-video generation, named AtomoVideo. Based on multi-granularity image injection, we achieve higher fidelity of the generated video to the given image. In addition, thanks to high quality datasets and training strategies, we achieve greater motion intensity while maintaining superior temporal consistency and stability. Our architecture extends flexibly to the video frame prediction task, enabling long sequence prediction through iterative generation. Furthermore, due to the design of adapter training, our approach can be well combined with existing personalised models and controllable modules.


We are thrilled to announce the release of the OmniACT dataset!  This revolutionary dataset and benchmark focuses on pushing the limits of how virtual agents can facilitate the automation of our computer tasks. Imagine less clicking and typing, and more observation as your computer takes care of tasks such as organizing schedules or arranging travel arrangements on its own.

Check it out ➡ [OmniACT Dataset on Hugging Face](https://huggingface.co/datasets/Writer/omniact)

For a deep dive, here’s the paper: [OmniACT Paper](https://arxiv.org/abs/2402.17553)

melisa osanseviero victor nbroad samusenps jeffboudier julien-c kiranr lhoestq pierrci nikjain clem NERDDISCO bk-ignite medmac01 thomwolf sbrandeis

femboysLover victor M1cler julien-c lhoestq clem bk-ignite medmac01 thomwolf


🔥 New multimodal leaderboard on the hub: ConTextual!

Many situations require models to parse images containing text: maps, web pages, real world pictures, memes, ... 🖼
So how do you evaluate performance on this task?

The ConTextual team introduced a brand new dataset of instructions and images, to test LMM (large multimodal models) reasoning capabilities, and an associated leaderboard (with a private test set).

This is super exciting imo because it has the potential to be a good benchmark both for multimodal models and for assistants' vision capabilities, thanks to the instructions in the dataset.

Congrats to @rohan598, @hbXNov, @kaiweichang and @violetpeng !!

Learn more in the blog: https://huggingface.co/blog/leaderboard-contextual
Leaderboard: https://huggingface.co/spaces/ucla-contextual/contextual_leaderboard

vladbogo BrigitteTousi Kukedlc osanseviero victor VictorSanh samusenps JasonZJ mvaloatto rohan598 puneetjindalisb whitew1994


VisionLLaMA: A Unified LLaMA Interface for Vision Tasks (2403.00522)

VisionLLaMA is a new vision transformer architecture that adapts the successful LLaMA language model design for vision tasks. By integrating components like rotary positional embeddings, SwiGLU activation, and LayerNorm from LLaMA, VisionLLaMA achieves very promising performance across various vision tasks, including image generation, classification, semantic segmentation, and object detection.

Keypoints:
* Outperforms state-of-the-art vision transformers like DiT, SiT, DeiT3, and Swin on multiple benchmarks and tasks.
* Leverages Auto-Scaled 2D Rotary Positional Embeddings (AS2DRoPE) to handle variable input resolutions efficiently.
* Serves as a powerful, unified modeling framework for vision generation and understanding tasks.

Paper: https://huggingface.co/papers/2403.00522
GitHub repo: https://github.com/Meituan-AutoML/VisionLLaMA

Congrats to the authors for their work!

clem jucamohedano osanseviero victor samusenps sgarbi MexIvanov trysem


Robust Speech Recognition via Large-Scale Weak Supervision (2212.04356)

Why does returning timestamps help Whisper reduce hallucinations? 🧐

Empirically, most practitioners have found that setting `return_timestamps=True` helps reduce hallucinations, particularly when doing long-form evaluation with Transformers’ “chunked” algorithm.

But why does this work?..

My interpretation is that forcing the model to predict timestamps is contradictory to hallucinations. Suppose you have the transcription:
```markdown
The cat sat on the on the on the mat.
```
Where we have a repeated hallucination for “on the”. If we ask the model to predict timestamps, then the “on the” has to contribute to the overall segment-level timing, e.g.:
```markdown
<|0.00|> The cat sat on the on the on the mat.<|5.02|>
```
However, it’s impossible to fit 3 copies of “on the” within the time allocation given to the segment, so the probability for this hallucinatory sequence becomes lower, and the model actually predicts the correct transcription with highest probability:
```markdown
<|0.00|> The cat sat on the mat.<|5.02|>
```
In this sense, the end timestamp is of the opposite of the initial timestamp constraint they describe in Section 4.5 of the paper https://huggingface.co/papers/2212.04356 → it helps the model remove extra words at the end of the sequence (rather than the initial timestamp which helps when the model ignores words at the start), but the overall principle is the same (using timestamps to improve the probability of more realistic sequences).

Leaving it open to you: why do you think timestamps reduces Whisper hallucinations?

clem abellion vladbogo daedaluswave osanseviero samusenps Csplk DRRRRDFFD on1onmangoes sanchit-gandhi jayzhuo michaelsymph EBalykov Mutturaj DattaKiran OjciecTadeusz ericlaycock dkakaie WIldOni ayoalfonso dzlandis peterciank


What if you could casually access your remote GPU in HF Spaces from the comfort of your local VSCode 🤯

ajibawa-2023 Gabriel victor mvaloatto medmac01 clem BrigitteTousi musfiqdehan osanseviero asusevski samusenps cnmoro mathiasn1 JackCloudman Nadav-Timor Felladrin muhtasham dwidlee peterschroedl not-lain radames

fffiloni taufiqdp medmac01 natolambert clem BrigitteTousi umuthopeyildirim osanseviero vtiyyal1 xxXWarMachineRoXxx muhtasham peterschroedl kramp not-lain seyf1elislam radames

tuanlda78202 medmac01 clem BrigitteTousi muhtasham peterschroedl not-lain


Huge new dataset for remote sensing on HF: Major TOM

More info in this thread https://x.com/mikonvergence/status/1764913256102031402?s=20

Note: video coming soon

mikonvergence abbasm2 clem osanseviero samusenps dillfrescott


The dataset of all Italian laws  just released https://huggingface.co/datasets/mii-llm/gazzetta-ufficiale

Kudos to @efederici @mferraretto @z-uo


The Stable Diffusion 3 research paper broken down, including some overlooked details! 📝

Model
📏 2 base model variants mentioned: 2B and 8B sizes

📐 New architecture in all abstraction levels:
- 🔽 UNet; ⬆ Multimodal Diffusion Transformer, bye cross attention 👋
- 🆕 Rectified flows for the diffusion process
- 🧩 Still a Latent Diffusion Model

📄 3 text-encoders: 2 CLIPs, one T5-XXL; plug-and-play: removing the larger one maintains competitiveness

🗃 Dataset was deduplicated with SSCD which helped with memorization (no more details about the dataset tho)

Variants
🔁 A DPO fine-tuned model showed great improvement in prompt understanding and aesthetics
✏ An Instruct Edit 2B model was trained, and learned how to do text-replacement

Results
✅ State of the art in automated evals for composition and prompt understanding
✅ Best win rate in human preference evaluation for prompt understanding, aesthetics and typography (missing some details on how many participants and the design of the experiment)

Paper: https://stabilityai-public-packages.s3.us-west-2.amazonaws.com/Stable+Diffusion+3+Paper.pdf

osanseviero fffiloni victor mvaloatto clem BrigitteTousi WaveCut EtienneDosSantos samusenps Tonic chethan62 Eungbean MrForExample Obhee GPT007

clem BrigitteTousi samusenps leegao19 Tonic Bils gonduras


Major TOM: Expandable Datasets for Earth Observation (2402.12095)

🗺 Major TOM: Expandable Datasets for Earth Observation

🚨 RECORD-BREAKING EO DATASET: the largest ever ML-ready Sentinel-2 dataset! It covers almost every single point on Earth captured by the Copernicus Sentinel-2 satellite. @mikonvergence and I are thrilled to finally announce the release of https://huggingface.co/datasets/Major-TOM/Core-S2L2A and https://huggingface.co/datasets/Major-TOM/Core-S2L1C

🌍 About half of the entire planet is covered. That's 2,245,886 patches of 1068 x 1068 pixels, available in both L1C and L2A. At 10 m resolution, we've got 256 million square km with over 2.5 trillion pixels. It's all yours with a few lines of code. See the paper linked below 🔽 for more info!

🧱 And this is just the beginning. We are currently preparing more datasets from different satellites for the Major TOM org. TOM stands for Terrestrial Observation Metaset - a simple set of rules for building an ecosystem of ML-ready EO datasets, which can be seamlessly combined as if they were Lego bricks.

🚴♀ Want to take the dataset for a spin? We have a viewer app on spaces that lets you go anywhere on Earth and shows you the data, if its available https://huggingface.co/spaces/Major-TOM/MajorTOM-Core-Viewer

📰 Preprint paper: https://huggingface.co/papers/2402.12095
💻 Colab example: https://colab.research.google.com/github/ESA-PhiLab/Major-TOM/blob/main/03-Filtering-in-Colab.ipynb

Thank you to the amazing 🤗Hugging Face team for the support on this one! @osanseviero @lhoestq @BrigitteTousi

mikonvergence lhoestq osanseviero julien-c taufiqdp mvaloatto clem BrigitteTousi nicolasdec kramp csaybar

lhoestq osanseviero julien-c clem BrigitteTousi nicolasdec mrapplegate samusenps brunosan


Just released moondream2 - a small 1.8B parameter vision language model. Now fully open source (Apache 2.0) so you can use it without restrictions on commercial use!

https://huggingface.co/vikhyatk/moondream2

osanseviero mrfakename andysalerno natolambert ajibawa-2023 Vigilence vignesh-fynd Tonic Csplk DmitryRyumin Chris4K chansung Dlbk Yoben fffiloni Kukedlc taufiqdp clem ostris samusenps saikatkumardey multimodalart veravira euclaise twenkid sbrandeis

osanseviero andysalerno OsnNos Vigilence vignesh-fynd Tonic kramp Dlbk sourceoftruthdata clem t1u1 damerajee


GaussianAvatar: Towards Realistic Human Avatar Modeling from a Single
  Video via Animatable 3D Gaussians (2312.02134)

🚀🎬🌟 New Research Alert - CVPR 2024! 🌟🎬 🚀
📄 Title: GaussianAvatar: Towards Realistic Human Avatar Modeling from a Single Video via Animatable 3D Gaussians 🌟🚀

👥 Authors: Liangxiao Hu et al.

📅 Conference: CVPR, Jun 17-21, 2024 | Seattle WA, USA 🇺🇸

🔗 Paper: https://huggingface.co/papers/2312.02134

🔗 Github Page: https://huliangxiao.github.io/GaussianAvatar
🔗 Repository: https://github.com/huliangxiao/GaussianAvatar

🔗 Video: https://www.youtube.com/watch?v=a4g8Z9nCF-k

📚 More Papers: more cutting-edge research presented at other conferences in the https://huggingface.co/spaces/DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin

🚀 Added to the Avatars Collection: https://huggingface.co/collections/DmitryRyumin/avatars-65df37cdf81fec13d4dbac36

🔍 Keywords: #GaussianAvatar #3DGaussians #HumanAvatarModeling #PoseDependentAppearance #DynamicAppearanceModeling #MotionEstimation #MonocularSettings #AppearanceQuality #RenderingEfficiency #CVPR2024 #DeepLearning #Animation #Innovation

DmitryRyumin vladbogo mickdarling clem MiSTe-R StrangeSX szymonrucinski Tonic sbrandeis


🌊 Released #LaVague, fullly open-source AI pipeline to turn natural language into browser actions!

In less than 150 lines of code (RAG with local embedding + Zephyr-7b-Gemma locally or Mixtral on HF Inference API), it generates #Selenium code from user query. In this GIF you can see it follow user instructions to command a browser to browse HF website!

Try it on Colab: colab.research.google.com/github/dhuynh95/LaVague/blob/main/LaVague.ipynb
GitHub: github.com/dhuynh95/LaVague

Pretty exciting how it becomes possible to create an AI assistant that could perform actions for us, such as logging on gov accounts, fill forms, or pull personal information!

It was quite fun to hack in the weekend using open-source tools, from @huggingface local embedding with transformers for local inference or HF Inference API, to RAG with @llama_index, through @MistralAI Mixtral model!

Some challenges: to make it run on Colab for the #GPU Poors, I first resorted to @huggingface Inference API with Mixtral as it was the only model good enough (gemma-7b did not make it and refused to produce code). But after some experimentations, I managed to make it work a local Zephyr-7b-Gemma so that people could run this assistant fully locally!

Because I used an off-the-shelf model, I had to improve performance with few-shot learning and Chain Of Thought, which managed to generate appropriate code!

I hope this project will herald a new dawn where transparent, private and local AI assistants help automate menial but critical tasks, such as helping fill taxes, book accomodation, or research information for us.

osanseviero kramp ejsellers vladbogo mexicanamerican 0xKrypt0r Dlbk qnguyen3 julien-c pranay-j victor diwank iamrobotbear samusenps bruceunx George-Blaze superLk timnon

osanseviero VictorSanh Guilherme34 Dlbk julien-c victor clem diwank bonsaielectric iamrobotbear samusenps z3ymd kgawron bruceunx Tavi31 superLk sbrandeis

fffiloni victor clem iamrobotbear Tavi31


Understand research papers easier with automatically generated Q&As by LLM (Gemini 1.0 Pro). For this purpose, I have built two projects.

- [Auto Paper Analysis](https://github.com/deep-diver/auto-paper-analysis) let you generate QAs on a list of papers. The paper list could be specified either from Hugging Face's Daily Papers or in a set of raw arXiv IDs. Then the generated QA dataset could be pushed to the Hugging Face Dataset. Refer to the attached image.

- [PaperQA Space application](https://huggingface.co/spaces/chansung/paper_qa) shows how to interact with the generated QA dataset. Search the paper by keyword or date, then understand it with the QAs (in ELI5 and technical versions). Check out the attached video, or visit the space directly.

This is a baby step for the automated paper analysis (summarization) to easily consume the exploding information in the field of AI. In the next phase, I am gonna need spend my time to enhance prompt engineering, UI/UX (such as Like/Dislike system), ...

However, in the meantime, I hope this project could be helpful for someone who struggles on understanding papers (new papers comes out even when I did finish reading a paper from yesterday yet,,)!

Also, any suggestion to improve this, please let me know :)

chansung Kukedlc medmac01 vladbogo arcdyn Lewdiculous pranay-j Srulikbd samusenps bh-lee akhaliq boapps genaiLLM sbrandeis


VisionLLaMA

A Unified LLaMA Interface for Vision Tasks

https://huggingface.co/papers/2403.00522

Large language models are built on top of a transformer-based architecture to process textual inputs. For example, the LLaMA stands out among many open-source implementations. Can the same transformer be used to process 2D images? In this paper, we answer this question by unveiling a LLaMA-like vision transformer in plain and pyramid forms, termed VisionLLaMA, which is tailored for this purpose. VisionLLaMA is a unified and generic modelling framework for solving most vision tasks. We extensively evaluate its effectiveness using typical pre-training paradigms in a good portion of downstream tasks of image perception and especially image generation. In many cases, VisionLLaMA have exhibited substantial gains over the previous state-of-the-art vision transformers. We believe that VisionLLaMA can serve as a strong new baseline model for vision generation and understanding.

osanseviero clem Kukedlc hasangoni vladbogo notune Yoben mvaloatto TD788432 samusenps Abecid jsfs11 Ryukijano sprime01 researchase sbrandeis


Today, we're launching an effort to empower the community to build impactful datasets collectively.

Good data is essential for the open-source AI community. Recently, Argilla and Hugging Face launched Data is Better Together. In less than two weeks, over 350 people ranked over 10k  prompts.

Today, we're shifting our focus to help support other community efforts to create datasets using Argilla and Hugging Face Spaces. This workflow means anyone with a Hugging Face account can contribute to a dataset in less than a minute.  We want to hear from anyone with ideas for creating important datasets as a community. This could include things like:

- Creating preference data for a language that lacks high-quality preference datasets.
- Building evaluation datasets for a new domain.
- Developing a dataset for a new task.

If you would like to get involved, join us in the `#data-is-better-together` Discord channel: https://discord.com/channels/879548962464493619/1205128865735770142.

You can read more in this blog post from @dvilasuero and I: https://huggingface.co/blog/community-datasets

osanseviero clem Kukedlc ajibawa-2023 danielus julien-c samusenps dillfrescott


🚨 Now you can run Starcoder- 2 models locally on your Mac M1 Pro Apple Silicon with 16GB memory! 🧑🏽💻 ⚡✨

Below is the UX with Twinny extension using https://huggingface.co/bigcode/starcoder2-3b for FIM and https://huggingface.co/codellama/CodeLlama-7b-Instruct-hf for chat. Dev tools is showing the prompt being sent to ollama server.

Starcoder-2 is now supported in `llama.cpp`  https://github.com/ggerganov/llama.cpp/pull/5795!
```
cd llama.cpp
python convert-hf-to-gguf.py ../starcoder2-3b/ --outfile models/starcoder2-3b.gguf --outtype "f16"
./quantize models/starcoder2-3b.gguf models/starcoder2-3b-Q4_K_M.gguf Q4_K_M
```
For more details, please go through the following tweet thread: https://x.com/sourab_m/status/1764583139798823235?s=20

osanseviero lvwerra clem Kukedlc notune jshuadvd codito rjmacarthy

clem Kukedlc Yoben BrigitteTousi rjmacarthy


AtP*: An efficient and scalable method for localizing LLM behaviour to
  components (2403.00745)

🔍 Today's pick in Interpretability & Analysis of LMs: AtP*: An efficient and scalable method for localizing LLM behaviour to components by J. Kramár T. Lieberum R. Shah @NeelNanda

The attribution patching method (AtP) can provide fast and effective approximations of activation patching, requiring only two forward passes and one backward pass to estimate the contribution of all network components for a given prompt pair.

While previous work highlighted the effectiveness of attribution patching, authors identify two settings leading to false negatives using AtP:

- When estimating the contribution of pre-activation components, if clean and noise inputs don’t lie in the same activation region, the first-order gradient approximation provided by the gradient leads to large errors (Fig 3).
- When the sum of direct and indirect effects is close to 0, even small approximation errors introduced by nonlinearities can greatly  affect the estimated contribution.

Authors propose two changes to the AtP method to mitigate such issues:

- recomputing the attention softmax for the selected component, and then taking a linear approximation to the remaining part of the model (QK Fix)
- Iteratively zeroing gradients at layers contributing to the indirect effects causing cancellation (GradDrop)

AtP and AtP* are compared across several patching settings for Pythia models, finding them effective while much less computationally expensive than other approaches. A new methodology is also proposed to estimate the magnitude of AtP* false negatives given a set of samples and desired confidence levels.

📄 Paper: https://huggingface.co/papers/2403.00745

🔍 All daily picks: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9

osanseviero NeelNanda clem Kukedlc julien-c nirmalendu01


Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (2402.19479)

Panda-70M is a new large-scale video dataset comprising 70 million high-quality video clips, each paired with textual captions, designed to be used as pre-training for video understanding tasks.

Key Points:
* Automatic Caption Generation: Utilizes an automatic pipeline with multiple cross-modality teacher models to generate captions for video clips.
* Fine-tuned Caption Selection: Employs a fine-tuned retrieval model to select the most appropriate caption from multiple candidates for each video clip.
* Improved Performance: Pre-training on Panda-70M shows significant performance gains in video captioning, text-video retrieval, and text-driven video generation.

Paper: https://huggingface.co/papers/2402.19479
Project page: https://snap-research.github.io/Panda-70M/
Code: https://github.com/snap-research/Panda-70M

Congrats to the authors @tschen, @aliaksandr-siarohin et al. for their work!

toanvu upleader ajibawa-2023 clem Kernel osanseviero dkgee ankur296 ChavyvAkvar tschen

clem osanseviero tschen nisten sbrandeis


Learning to Generate Instruction Tuning Datasets for Zero-Shot Task
  Adaptation (2402.18334)

Diaries of Open Source. Part 1.

What a week! Here are some of the exciting Open Source releases of the week!

1. BigCode releases The Stack v2 and StarCoder 2
Resources in https://huggingface.co/posts/loubnabnl/596860170283496
Blog https://huggingface.co/blog/starcoder2
Collection: https://huggingface.co/collections/bigcode/starcoder2-65de6da6e87db3383572be1a

2. Playground v2.5, a very powerful new text-to-image model
Model: https://huggingface.co/playgroundai/playground-v2.5-1024px-aesthetic
Demo: https://huggingface.co/spaces/playgroundai/playground-v2.5
Blog: https://playground.com/blog/playground-v2-5

3.Evo: DNA foundation models
Blog: https://arcinstitute.org/news/blog/evo
Models: https://huggingface.co/togethercomputer/evo-1-131k-base

4. OpenHermesPreferences: a dataset of ~1 million AI Preferences https://huggingface.co/datasets/argilla/OpenHermesPreferences

5. SpeechBrain 1.0: a toolkit with hundreds of recipes and pretrained models for audio-related tasks, such as speech recognition, diarization, and enhancement. New major release!
HF repos: https://huggingface.co/speechbrain
Website: https://speechbrain.github.io/

6. Tower: a suite of Llama-based multilingual translation models  https://huggingface.co/collections/Unbabel/tower-659eaedfe36e6dd29eb1805c

7. AllenAI releases OLMo-7B-Instruct
 https://huggingface.co/collections/allenai/olmo-suite-65aeaae8fe5b6b2122b46778

8. DIBT - An crowdsourced effort to human-rate prompts. Its 10k prompts dataset is released ttps://huggingface.co/datasets/DIBT/10k_prompts_ranked

9. ChatMusician: A Llama 2 fine-tuned model for music generation  https://huggingface.co/m-a-p/ChatMusician

10. Bonito, an model that converts data into synthetic instruction datasets
GitHub: https://github.com/BatsResearch/bonito
Model: https://huggingface.co/BatsResearch/bonito-v1
Paper: https://huggingface.co/papers/2402.18334

DmitryRyumin vladbogo clem chansung Kernel kramp tofuCheng tomaarsen lvwerra dkgee not-lain julien-c mvaloatto


MagicAnimate: Temporally Consistent Human Image Animation using
  Diffusion Model (2311.16498)

🚀💃🌟 New Research Alert - CVPR 2024! 🌟🕺 🚀
📄 Title: MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model 🌟🚀

👥 Authors: @junhao910323, @hansyan et al.

📅 Conference: CVPR, Jun 17-21, 2024 | Seattle WA, USA 🇺🇸

🤗 Demo: https://huggingface.co/spaces/zcxu-eric/magicanimate

🔗 Paper: https://huggingface.co/papers/2311.16498
🔗 Github Page: https://showlab.github.io/magicanimate/
🔗 Repository: https://github.com/magic-research/magic-animate

🔥 Model 🤖: https://huggingface.co/zcxu-eric/MagicAnimate

📚 More Papers: more cutting-edge research presented at other conferences in the https://huggingface.co/spaces/DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin

🚀 Added to the Avatars Collection: https://huggingface.co/collections/DmitryRyumin/avatars-65df37cdf81fec13d4dbac36

🔍 Keywords: #MagicAnimate #DiffusionModel #HumanImageAnimation #CVPR2024 #Diffusion #DeepLearning #Innovation

DmitryRyumin musfiqdehan pkedzia vladbogo osanseviero victor toanvu clem samusenps sbrandeis


What Evidence Do Language Models Find Convincing? (2402.11782)

"What Evidence Do Language Models Find Convincing?" is a new paper that explores what types of evidence and argumentation techniques language models find convincing when presented with ambiguous, open-domain questions that have conflicting answers online.

Keypoints:
* Dataset: It introduces "ConflictingQA," a dataset of controversial questions and real-world evidence paragraphs supporting both "yes" and "no" answers.
* Convincingness Metric: It uses the "paragraph win rate" - when shown two conflicting paragraphs, this measures how often a model predicts the answer that aligns with a given paragraph's stance.
* Current models rely on the relevance of the content to the query, while largely ignoring stylistic features such as whether a text contains scientific references or if it is written with a neutral tone.

Congrats to the authors for their work!

Paper: https://huggingface.co/papers/2402.11782
Code: https://github.com/AlexWan0/rag-convincingness

samusenps dillfrescott osanseviero gsarti victor clem Kukedlc victorshenbr sbrandeis


Introducing: Zephyr Gemma!

The community has struggled to do a good preference-tune of Gemma, so the amazing @lewtun and @philschmid built an open-source recipe and trained a model to help people get started.

Handbook: https://github.com/huggingface/alignment-handbook/blob/main/recipes/zephyr-7b-gemma/README.md
Model: https://huggingface.co/HuggingFaceH4/zephyr-7b-gemma-v0.1
Demo: https://huggingface.co/spaces/HuggingFaceH4/zephyr-7b-gemma-chat

Some interesting details
- Fine-tuned on DEITA and DPOed with Argilla DPO dataset
- Very strong MT Bench results (7.81), better than Zephyr Beta (mistral based) and Gemma Instruct
- Can run locally with tools such as llama.cpp on a Mac
- Not so good AGIEval results compared to mistral-based tunes
- All training code is open-sourced
- Trained for 105 minutes on 8x H100
- No system message

Big kudos to the team! Super exciting to see a good fine-tune for Gemma

yjernite vladbogo clem samusenps dkyazze taufiqdp pdina


It feels awkward having my first post sharing my stuff, but this is a weekend project that I really enjoyed working on. I'd love to meet more people interested in random ideas like this.

A hard part of building AI applications is choosing which model to use. What if we don’t have to? What if we can predict the best model for any prompt?

Predictive human preference aims to predict which model users might prefer for a specific query.

https://huyenchip.com/2024/02/28/predictive-human-preference.html

One use case is model routing. If we know in advance that for a prompt, users will prefer Claude Instant’s response over GPT-4, and Claude Instant is cheaper/faster than GPT-4, we can route this prompt to Claude Instant. Model routing has the potential to increase response quality while reducing costs and latency.

One pattern is that for simple prompts, weak models can do (nearly) as well as strong models. For more challenging prompts, however, users are more likely to prefer stronger models. Here’s a visualization of predicted human preference for an easy prompt (“hello, how are you?”) and a challenging prompt (“Explain why Planc length …”).

Preference predictors make it possible to create leaderboards unique to any prompt and domain.

clem Nganx andrewrreed vladbogo osanseviero loubnabnl EmilyWitko ankity09 samusenps callmesan bedestaz ChavyvAkvar dortorwu2 gsarti Shaheer-ipynb aotrih ntdas dark-pen julien-c kgourgou pcuenq valeriiakuka sbrandeis

andrewrreed mvaloatto kramp julien-c kgourgou gigant Bkarine


👷🏽♀📚🔨  Announcing the Foundation Model Development Cheatsheet!

My first 🤗Post🤗 ever to announce the release of a fantastic collaborative resource to support model developers across the full development stack: The FM Development Cheatsheet available here:  https://fmcheatsheet.org/

The cheatsheet is a growing database of the many crucial resources coming from open research and development efforts to support the responsible development of models. This new resource highlights essential yet often underutilized tools in order to make it as easy as possible for developers to adopt best practices, covering among other aspects:
🧑🏼🤝🧑🏼 data selection, curation, and governance;
📖 accurate and limitations-aware documentation;
⚡ energy efficiency throughout the training phase;
📊 thorough capability assessments and risk evaluations;
🌏 environmentally and socially conscious deployment strategies.

We strongly encourage developers working on creating and improving models to make full use of the tools listed here, and to help keep the resource up to date by adding the resources that you yourself have developed or found useful in your own practice 🤗

Congrats to all the participants in this effort for the release! Read more about it from:
@Shayne - https://twitter.com/ShayneRedford/status/1763215814860186005
@hails and @stellaathena - https://blog.eleuther.ai/fm-dev-cheatsheet/
@alon-albalak - http://nlp.cs.ucsb.edu/blog/a-new-guide-for-the-responsible-development-of-foundation-models.html

And also to  @gabrielilharco @sayashk @kklyman @kylel @mbrauh @fauxneticien @avi-skowron @Bertievidgen Laura Weidinger, Arvind Narayanan, @VictorSanh @Davlan @percyliang Rishi Bommasani, @breakend @sasha 🔥

VictorSanh hails mohammedbriman Shayne kklyman pietrolesci osanseviero sayhan poliva loubnabnl yongzx evdcush alon-albalak boapps mbrauh Epiculous BrigitteTousi mathiasn1 sasha de-Rodrigo alielfilali01

hails mohammedbriman Shayne kklyman mvaloatto osanseviero evdcush alon-albalak Epiculous BrigitteTousi


Defending LLMs against Jailbreaking Attacks via Backtranslation (2402.16459)

https://huggingface.co/papers/2402.16459
**Defending LLMs against Jailbreaking Attacks via Backtranslation**

I really love this! Its a really innovative way to get robust defense against jailbreaking. Its not cheap, 2-3 calls per user request. But for some use-cases it can be worth it!


8 Spaces Of The Week is nice, but 840 is even better! 🔥

Here is the complete library of ALL Spaces featured by Hugging Face since October 2021:

All Spaces Of The Week - https://huggingface.co/spaces/mvaloatto/ASOTW

-
A special mention goes to @osanseviero, whose collection inspired me to design this dedicated Space. Another shoutout to @victor, whose intricately designed Spaces cards motivated me to step up my CSS game :) I plan to release additional features in the future. In the meantime, suggestions are welcome!

osanseviero vladbogo hysts victor kramp clem samusenps fffiloni nbroad Nymbo


An increasing number of engineers and researchers are developing foundational models. Navigating the tools, resources, codebases, and best practices guides is daunting for new contributors.

Introducing the Foundation Model Development Cheatsheet, a succinct guide with 250+ resources & tools for:
📖 sourcing data
🔍 documenting & audits
🌍 environmental impact
🥊 risks & harms eval
🎮 release & monitoring

https://fmcheatsheet.org/

👐 What tools & resources should appear in that cheatsheet? Contributions encouraged!

This is the result of a large collaboration between many organizations promoting open-science, and spearheaded by @Shayne 🔥

Shayne osanseviero kirch mayacinka mvaloatto ajibawa-2023 FranzNo yjernite erikab samusenps dkyazze sbarman25 victor lixinjie


Training a SOTA code LLM with a fully transparent library (nanotron) built from scratch -> Done ✅

⭐  BlogPost: https://huggingface.co/blog/starcoder2
🔗 StarCoder2 collection: https://huggingface.co/collections/bigcode/starcoder2-65de6da6e87db3383572be1a


🙋🏻♂ hey there folks ,

Star coder came out and it's really fascinating in more ways than one !

first off it codes well already. but secondly it's reported to "know" 101 programming languages !

that actually means it's ripe for fine tunes, so if you're like me you've been bookmarking cool datasets and cant wait to get started !

that said , here's a cool demo where you can try it out now : https://huggingface.co/spaces/Tonic/starcoder2

turns out it can program a T5 demo using gradio !

victor osanseviero kgourgou clem fffiloni

mvaloatto ajibawa-2023 clem goncharenko yxxsgdmn


Where I work, we are obsessed with what happens to a model's performance after it has been deployed. We call this post-deployment data science.

Let me tell you about a post-deployment data science algorithm that we recently developed to measure the impact of Concept Drift on a model's performance.

How can we detect Concept Drift? 🤔

All ML models are designed to do one thing: learning a probability distribution in the form of P(y|X). In other words, they try to learn how to model an outcome 'y' given the input variables 'X'. 🧠

This probability distribution, P(y|X), is also called Concept. Therefore, if the Concept changes, the model may become invalid.

❓But how do we know if there is a new Concept in our data?
❓Or, more important, how do we measure if the new Concept is affecting the model's performance?

💡 We came up with a clever solution where the main ingredients are a reference dataset, one where the model's performance is known, and a dataset with the latest data we would like to monitor.

👣 Step-by-Step solution:

1⃣ We start by training an internal model on a chunk of the latest data. ➡ This allows us to learn the new possible Concept presented in the data.

2⃣ Next, we use the internal model to make predictions on the reference dataset.

3⃣ We then estimate the model's performance on the reference dataset, assuming the model's predictions on the monitoring data as ground truth.

4⃣ If the estimated performance of the internal model and the actual monitored model are very different, we then say that there has been a Concept Drift.

To quantify how this Concept impacts performance, we subtract the actual model's performance on reference from the estimated performance and report a delta of the performance metric. ➡ This is what the plot below shows. The change of the F1-score due to Concept drift! 🚨

This process is repeated for every new chunk of data that we get. 🔁

ajibawa-2023 fblgit jessicagab Nhebo victor vishwask codito anujd9


Real-time object detection w/ 🤗 Transformers.js, running YOLOv9 locally in your browser! 🤯

Try it out yourself: https://huggingface.co/spaces/Xenova/video-object-detection
(Model used + example code: https://huggingface.co/Xenova/gelan-c_all)

This demo shows why on-device ML is so important:
1. Privacy - local inference means no user data is sent to the cloud
2. No server latency - empowers developers to build real-time applications
3. Lower costs - no need to pay for bandwidth and processing of streamed video

I can't wait to see what you build with it! 🔥

DmitryRyumin macadeliccc giux78 radames ajibawa-2023 osanseviero Dlbk jaickerag victor mvaloatto clem samusenps bwang0911 Thepickledegg hogunkim felixdrp Noomam Youngwon elcrei


I have dedicated several days, working over 12 hours each day, on SUPIR (Scaling-UP Image Restoration), a cutting-edge image enhancement and upscaling model introduced in the paper Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild.

This model is simply mind-blowing. At the bottom of this post, you will see side-by-side comparisons of SUPIR versus the extremely expensive online service, Magnific AI. Magnific is known to be the best among the community. However, SUPIR is by far superior. SUPIR also significantly outperforms Topaz AI upscale. SUPIR manages to remain faithful to the original image almost 100% while adding details and achieving super upscaling with the best realism.

You can read the full blog post here : https://huggingface.co/blog/MonsterMMORPG/supir-sota-image-upscale-better-than-magnific-ai

ameerazam08 clem iDrops samusenps diogofranciscop yxxsgdmn Johnnycadelover


🔥 What's that biomedical model that got 170,763 downloads last month on HuggingFace?! Well, the paper is finally published! #BioLORD

📰 Read our article in the Journal of the American Medical Informatics Association:
https://academic.oup.com/jamia/advance-article/doi/10.1093/jamia/ocae029/7614965

📝`TLDR:` BioLORD-2023 is a series of semantic language models for the biomedical domain, capable of representing clinical concepts and sentences in a semantic space aligned with human preferences. Our new multilingual version supports 50+ languages and is further finetuned on 7 European languages. These models were trained contrastively and through distillations, using a corpus unifying in the same latent space the concept names of biomedical concepts and their descriptions. For concepts which didn't have a description written by humans in UMLS, we use information contained in the SnomedCT knowledge graph and the capabilities of ChatGPT to generate synthetic data and improve our results.

🤗 Access our models from the HuggingFace hub, including the new 2023-C and 2023-S variants:
https://huggingface.co/FremyCompany/BioLORD-2023
https://huggingface.co/FremyCompany/BioLORD-2023-M
https://huggingface.co/FremyCompany/BioLORD-2023-S
https://huggingface.co/FremyCompany/BioLORD-2023-C

ilGawo cdevelder osanseviero katielink julien-c clem macadeliccc UMCU danielhanchen samusenps yxxsgdmn pdelobelle


The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (2402.17764)

The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

https://huggingface.co/papers/2402.17764

Recent research, such as BitNet, is paving the way for a new era of 1-bit Large Language Models (LLMs). In this work, we introduce a 1-bit LLM variant, namely BitNet b1.58, in which every single parameter (or weight) of the LLM is ternary {-1, 0, 1}. It matches the full-precision (i.e., FP16 or BF16) Transformer LLM with the same model size and training tokens in terms of both perplexity and end-task performance, while being significantly more cost-effective in terms of latency, memory, throughput, and energy consumption. More profoundly, the 1.58-bit LLM defines a new scaling law and recipe for training new generations of LLMs that are both high-performance and cost-effective. Furthermore, it enables a new computation paradigm and opens the door for designing specific hardware optimized for 1-bit LLMs.

Azamat1k bofenghuang osanseviero clem macadeliccc ssone95 minjejeon SRDdev arcdyn ajibawa-2023 Kukedlc yongzx eramax mvaloatto kgourgou danielhanchen arjunsriva goncharenko notune koflerdavid krishnapraveen Banshal Hemanth-thunder Aishou northern-64bit tuanlda78202 andrewrreed CarlLee rippertnt joecrypto feveromo mathiasn1 RachidAR SaiNikhileshReddy yxxsgdmn julien-c femboysLover victor wath5

clem macadeliccc giux78 ssone95 fblgit adhisetiawan tuanlda78202 samusenps mindrage Parth SaiNikhileshReddy julien-c ncard


⭐ Today we’re releasing The Stack v2 & StarCoder2:  a series of 3B, 7B & 15B code generation models trained on 3.3 to 4.5 trillion tokens of code:

- StarCoder2-15B matches or outperforms CodeLlama 34B, and approaches DeepSeek-33B on multiple benchmarks.
- StarCoder2-3B outperforms StarCoderBase-15B and similar sized models.
- The Stack v2 a 4x larger dataset than the Stack v1, resulting in 900B unique code tokens 🚀
As always, we released everything from models and datasets to curation code. Enjoy!

🔗  StarCoder2 collection: https://huggingface.co/collections/bigcode/starcoder2-65de6da6e87db3383572be1a
🔗  Paper: https://drive.google.com/file/d/17iGn3c-sYNiLyRSY-A85QOzgzGnGiVI3/view
🔗  BlogPost: https://huggingface.co/blog/starcoder2
🔗  Code Leaderboard: https://huggingface.co/spaces/bigcode/bigcode-models-leaderboard

lvwerra DmitryRyumin thomwolf olivierdehaene smangrul osanseviero mcpotato alielfilali01 SivilTaram victor yuxiang630 BrigitteTousi VictorSanh harmdevries RaymondLi 3outeille Azamat1k vladbogo bofenghuang ncoop57 philschmid Avremi clem admarcosai macadeliccc mayank-mishra hysts kramp nouamanetazi danielhanchen goncharenko MoritzLaurer tbeck not-lain amyeroberts andrewrreed yxxsgdmn seyf1elislam omaryshchenko mishig

lvwerra thomwolf olivierdehaene smangrul osanseviero mcpotato alielfilali01 SivilTaram victor yuxiang630 BrigitteTousi VictorSanh harmdevries RaymondLi 3outeille ncoop57 philschmid clem admarcosai macadeliccc EdoAbati euclaise ajibawa-2023 mayank-mishra dalraf nouamanetazi tbeck not-lain amyeroberts andrewrreed samusenps seyf1elislam arjunguha mishig Ramikan-BR

lvwerra thomwolf olivierdehaene smangrul osanseviero SivilTaram victor yuxiang630 BrigitteTousi RaymondLi 3outeille hlarcher OmBenz philschmid Abderrazak clem admarcosai mayank-mishra not-lain mishig


EMO: Emote Portrait Alive - Generating Expressive Portrait Videos with
  Audio2Video Diffusion Model under Weak Conditions (2402.17485)

🌟🎭✨ Exciting News! The Latest in Expressive Video Portrait Generation! 🌟🎭✨

📄 Title: EMO: Emote Portrait Alive - Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions

👥 Authors: Linrui Tian, @lucaskingjade, Bang Zhang, and @Liefeng

🔗 Paper: https://huggingface.co/papers/2402.17485
🔗 Github Page: https://humanaigc.github.io/emote-portrait-alive
🔗 Repository: https://github.com/HumanAIGC/EMO

🔍 Keywords: #EMO #EmotePortrait #Audio2VideoDiffusion #ExpressiveAnimations #VideoGeneration  #DigitalArt #HumanExpression #ComputerVision #DeepLearning #AI

🚀 Added to the Avatars Collection: https://huggingface.co/collections/DmitryRyumin/avatars-65df37cdf81fec13d4dbac36

s3nh osanseviero DmitryRyumin Azamat1k aloobun sachaarbonel plmsmile ajibawa-2023 victor mvaloatto Hemanth-thunder samusenps CedrickChu thomwolf newxuyangcao oviniciusfeitosa


Super excited to share with you all our latest contribution from 2A2I.

Today we announce : https://huggingface.co/datasets/2A2I/Arabic-OpenHermes-2.5

Arabic-OpenHermes-2.5 Is simply the translation of the original dataset released by @teknium couple months ago !
In fact it looks as a simple task ! In reality it was quite a laborious job !
But thanks to @maghwa & @medmac01 this dataset managed to see the light today and help creating better  / more aligned arabic LLMs in the near future.

If you are interested to join us and/or help us, please leave a comment below or visit our HuggingFace Org Card for more details about How/What you can do.

More datasets to come and more models are in the way 🔥

medmac01 NovoCode osanseviero ajibawa-2023 maghwa mvaloatto sayhan samusenps


🚨 New Release of 🤗PEFT!

1. New methods for merging LoRA weights. Refer this HF Post for more details: https://huggingface.co/posts/smangrul/850816632583824

2. AWQ and AQLM support for LoRA. You can now:
- Train adapters on top of 2-bit quantized models with AQLM
- Train adapters on top of powerful AWQ quantized models
Note for inference you can't merge the LoRA weights into the base model!

3. DoRA support: Enabling DoRA is as easy as adding `use_dora=True` to your `LoraConfig`. Find out more about this method here: https://arxiv.org/abs/2402.09353

4. Improved documentation, particularly docs regarding PEFT LoRA+DeepSpeed and PEFT LoRA+FSDP! 📄 Check out the docs at https://huggingface.co/docs/peft/index.

5. Full Release Notes: https://github.com/huggingface/peft/releases/tag/v0.9.0

osanseviero Kukedlc clem Ji-Ha samusenps alielfilali01 mudogruer ybelkada ryokeken Cstark dahwinsingularity

NickyNicky Jenish-23 ixaxaar ababio rizwan-ai ybelkada Cstark ZennyKenny catastropiyush


🆕 There's now a visible description on Spaces previews. You can set the description of your Spaces by editing their README.md - it should make your Spaces more discoverable 🚀.

Felladrin osanseviero mvaloatto clefourrier alielfilali01 clem Vokturz samusenps ojasvisingh786


CausalGym: Benchmarking causal interpretability methods on linguistic
  tasks (2402.12560)

🔍 Today's pick in Interpretability & Analysis of LMs: CausalGym: Benchmarking causal interpretability methods on linguistic tasks by @aryaman D. Jurafsky @cgpotts

TL;DR: Introduce a revisited benchmark to evaluate the effectiveness and reliability of intervention methods across several linguistic phenomena.

While several interpretability methods are currently used to discover task-relevant model components, their performance and reliability is seldom tested broadly.

This paper adapts the SyntaxGym benchmark, originally conceived for the study of psycholinguistic phenomena such as subject-verb agreement and garden-path sentences, to evaluate intervention-based interpretability methods. In practice, faithful interventions over model components are expected to cause a predictable change in model prediction (e.g. singular -> plural verb).

Various methods are benchmarked on Pythia models ranging from 14M to 6.9B params, finding Distributed Alignment Search (DAS) to consistently outperform other approaches, followed by probing. When recurring to control tasks to account for the expressivity of supervised methods, probing is found to be more reliable than DAS in larger model sizes.

Authors conclude with an evaluation of how features driving linguistically plausible behaviours emerge during model training. These features are observed to emerge in Pythia models after 1k training steps, and become progressively more complex over time.

📄 Paper: https://huggingface.co/papers/2402.12560
💻 Code: https://github.com/aryamanarora/causalgym
🔡 Dataset: https://huggingface.co/datasets/aryaman/causalgym

🔍 All daily picks: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9


ChatMusician: Understanding and Generating Music Intrinsically with LLM (2402.16153)

ChatMusician

Understanding and Generating Music Intrinsically with LLM

https://huggingface.co/papers/2402.16153

While Large Language Models (LLMs) demonstrate impressive capabilities in text generation, we find that their ability has yet to be generalized to music, humanity's creative language. We introduce ChatMusician, an open-source LLM that integrates intrinsic musical abilities. It is based on continual pre-training and finetuning LLaMA2 on a text-compatible music representation, ABC notation, and the music is treated as a second language. ChatMusician can understand and generate music with a pure text tokenizer without any external multi-modal neural structures or tokenizers. Interestingly, endowing musical abilities does not harm language abilities, even achieving a slightly higher MMLU score. Our model is capable of composing well-structured, full-length music, conditioned on texts, chords, melodies, motifs, musical forms, etc, surpassing GPT-4 baseline. On our meticulously curated college-level music understanding benchmark, MusicTheoryBench, ChatMusician surpasses LLaMA2 and GPT-3.5 on zero-shot setting by a noticeable margin. Our work reveals that LLMs can be an excellent compressor for music, but there remains significant territory to be conquered. We release our 4B token music-language corpora MusicPile, the collected MusicTheoryBench, code, model and demo in GitHub.

regikono euclaise mexicanamerican ajibawa-2023 mvaloatto danielhanchen korhancagla osanseviero kramp victor Azamat1k Ranjanj1 Hoctar77 rafaelpierrehf

clem danielhanchen osanseviero EquinoxElahin asus4


Gemma QLoRA finetuning is now 2.4x faster and uses 58% less VRAM than FA2 through 🦥Unsloth! Had to rewrite our Cross Entropy Loss kernels to work on all vocab sizes, re-design our manual autograd engine to accept all activation functions, and more! I wrote all about our learnings in our blog post: https://unsloth.ai/blog/gemma.

Also have a Colab notebook with no OOMs, and has 2x faster inference for Gemma & how to merge and save to llama.cpp GGUF & vLLM: https://colab.research.google.com/drive/10NbwlsRChbma1v55m8LAPYG15uQv6HLo?usp=sharing

And uploaded 4bit pre-quantized versions for Gemma 2b and 7b: https://huggingface.co/unsloth/gemma-7b-bnb-4bit https://huggingface.co/unsloth/gemma-2b-bnb-4bit

```
from unsloth import FastLanguageModel
model, tokenzer = FastLanguageModel.from_pretrained("unsloth/gemma-7b")
model = FastLanguageModel.get_peft_model(model)
```

Sylvestre harpreetsahota cnmoro clem osanseviero ybelkada rbiswasfc alielfilali01 danielus ambivalent02 victor


The open-source AI community can build impactful datasets collectively!

Announcing https://huggingface.co/datasets/DIBT/10k_prompts_ranked, the first dataset release from Data Is Better Together.

Created in <2 weeks by the community. Includes:

✨ 10,000+ prompt quality ratings
🧑💻 Human and synthetic data prompts
🌐 Generated by 300+ contributors

How and why collaborative datasets?

It's no secret that high-quality open data is essential for creating better open models. The open source community shares 100s of models, datasets and demos openly weekly, but collectively building open datasets has been less explored.

Datasets have a massive role in shaping what models can be created. If we want more high-quality models for all languages, domains and tasks, we need more and better open datasets for all languages, domains and tasks!

To explore how the community could build impactful datasets collectively, Argilla added support for HF authentication for Argilla instances hosted on a Hugging Face Space. Anyone with an HF login could begin contributing to a dataset in <1 minute.

To test this new workflow, we launched a task to rank the quality of prompts (human and synthetically generated).

In less than two weeks, we built a community of over 300 contributors for this dataset 🤗

This dataset became a reality thanks to the dedication of all the individuals who lent their support ❤ To see the amazing people behind this dataset, visit https://huggingface.co/spaces/DIBT/prompt-collective-dashboard

This is just the start for collectively building powerful open datasets!

osanseviero alvarobartt dvilasuero samusenps clem TuringsSolutions danielhanchen alielfilali01 mmhamdy Tonic

victor osanseviero alvarobartt dvilasuero DmitryRyumin Sylvestre clem danielhanchen alielfilali01


🚀 Just released version 0.21.0 of the `huggingface_hub` Python library!

Exciting updates include:
🖇 Dataclasses everywhere for improved developer experience!
💾 HfFileSystem optimizations!
🧩 `PyTorchHubMixin` now supports configs and safetensors!
✨ `audio-to-audio` supported in the InferenceClient!
📚 Translated docs in Simplified Chinese and French!
💔 Breaking changes: simplified API for listing models and datasets!

Check out the full release notes for more details: https://huggingface.co/spaces/Wauplin/huggingface_hub/discussions/4 🤖💻

DmitryRyumin bmuskalla pcuenq osanseviero lunarflu clem severo pierrci harpreetsahota digiplay samusenps ajibawa-2023 dillfrescott MoritzLaurer dalraf victor mvaloatto yjernite danielus Kvikontent tonyassi radames

alvarobartt osanseviero lunarflu clem pierrci harpreetsahota dillfrescott yjernite radames

fffiloni kramp harpreetsahota dillfrescott radames


I've tried DoRA (https://arxiv.org/abs/2402.09353) with SDXL using PEFT, outputs are quite detailed 🤩🌟
as usual trained on lego dataset I compiled, I compared them with previously trained pivotal tuned model and the normal DreamBooth model before that 😊

Notebook by @linoyts https://colab.research.google.com/drive/134mt7bCMKtCYyYzETfEGKXT1J6J50ydT?usp=sharing
Integration to PEFT by @BenjaminB https://github.com/huggingface/peft/pull/1474 (more info in the PR)

jeanflop vladbogo NeuralKartMocker davideuler osanseviero Falah harpreetsahota jucamohedano ajibawa-2023 terpjwu1 CarlLee mvaloatto danielhanchen smangrul adamelliotfields fffiloni rafaelpierrehf sgch2023 radames mathiasn1 julien-c

jeanflop Falah danielhanchen smangrul rafaelpierrehf


FuseChat: Knowledge Fusion of Chat Models (2402.16107)

🚀🔥🌟 New Research Alert - ICLR 2024! 🌟🔥🚀
📄 Title: FuseChat: Revolutionizing Chat Models Fusion 🌟🚀

👥 Authors: @Wanfq, @passerqxj et al.

📅 Conference: ICLR, May 7-11, 2024 | Vienna, Austria 🇦🇹

🔗 Paper: https://huggingface.co/papers/2402.16107
🔗 Repository: https://github.com/fanqiwan/FuseLLM

🔥 Models 🤖:
1⃣ FuseChat-7B-VaRM: https://huggingface.co/FuseAI/FuseChat-7B-VaRM
2⃣ FuseChat-7B-Slerp: https://huggingface.co/FuseAI/FuseChat-7B-Slerp
3⃣ OpenChat-3.5-7B-Solar: https://huggingface.co/FuseAI/OpenChat-3.5-7B-Solar
4⃣ FuseChat-7B-TA: https://huggingface.co/FuseAI/FuseChat-7B-TA
5⃣ OpenChat-3.5-7B-Mixtral: https://huggingface.co/FuseAI/OpenChat-3.5-7B-Mixtral

📚 More Papers: more cutting-edge research presented at other conferences in the https://huggingface.co/spaces/DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin

🔍 Keywords: #FuseChat #ChatModels #KnowledgeFusion #ICLR2024 #AI #Innovation #FuseLLM

osanseviero tofuCheng vtiyyal1 Hoioi danielhanchen Concor sbrandeis


Genie: Generative Interactive Environments (2402.15391)

Genie is a new method from Google DeepMind that generates interactive, action-controllable virtual worlds from unlabelled internet videos using.

Keypoints:
* Genie leverages a spatiotemporal video tokenizer, an autoregressive dynamics model, and a latent action model to generate controllable video environments.
* The model is trained on video data alone, without requiring action labels, using unsupervised learning to infer latent actions between frames.
* The method restricts the size of the action vocabulary to 8 to ensure that the number of possible latent actions remains small.
* The dataset used for training is generated by filtering publicly available internet videos with specific criteria related to 2D platformer games for a total of 6.8M videos used for training.

Paper: https://huggingface.co/papers/2402.15391
Project page: https://sites.google.com/view/genie-2024/
More detailed overview in my blog: https://huggingface.co/blog/vladbogo/genie-generative-interactive-environments

Congrats to the authors for their work!

jsunster merve osanseviero kramp terpjwu1 programORdie2 mvaloatto danielhanchen Concor 2dts notune thomwolf julien-c


https://huggingface.co/google/gemma-7b-it is super good!

I wasn't convinced at first, but after vibe-checking it...I'm quite impressed.

I've got a notebook here, which is kind of a framework for vibe-checking LLMs.

In this notebook, I take Gemma for a spin on a variety of prompts:
•  [nonsensical tokens](https://huggingface.co/datasets/harpreetsahota/diverse-token-sampler]
• [conversation where I try to get some PII)(https://huggingface.co/datasets/harpreetsahota/red-team-prompts-questions)
• [summarization ability](https://huggingface.co/datasets/lighteval/summarization)
• [instruction following](https://huggingface.co/datasets/harpreetsahota/Instruction-Following-Evaluation-for-Large-Language-Models]
• [chain of thought reasoning](https://huggingface.co/datasets/ssbuild/alaca_chain-of-thought)

I then used LangChain evaluators (GPT-4 as judge), and track everything in LangSmith. I made public links to the traces where you can inspect the runs.

I hope you find this helpful, and I am certainly open to feedback, criticisms, or ways to improve.

Cheers:

You can find the notebook here: https://colab.research.google.com/drive/1RHzg0FD46kKbiGfTdZw9Fo-DqWzajuoi?usp=sharing

clem mvaloatto lunarflu osanseviero diwank harpreetsahota danielhanchen eramax arjunpatel adamelliotfields sbarman25 pansapiens


🚨 We’ve just released a new tool to compare the performance of models in the 🤗 Open LLM Leaderboard: the Comparator 🎉
https://huggingface.co/spaces/open-llm-leaderboard/comparator

Want to see how two different versions of LLaMA stack up? Let’s walk through a step-by-step comparison of LLaMA-3.1 and LLaMA-3.2. 🦙🧵👇

1/ Load the Models' Results
- Go to the 🤗 Open LLM Leaderboard Comparator: https://huggingface.co/spaces/open-llm-leaderboard/comparator
- Search for "LLaMA-3.1" and "LLaMA-3.2" in the model dropdowns.
- Press the Load button. Ready to dive into the results!

2/ Compare Metric Results in the Results Tab 📊
- Head over to the Results tab.
- Here, you’ll see the performance metrics for each model, beautifully color-coded using a gradient to highlight performance differences: greener is better! 🌟
- Want to focus on a specific task? Use the Task filter to hone in on comparisons for tasks like BBH or MMLU-Pro.

3/ Check Config Alignment in the Configs Tab ⚙
- To ensure you’re comparing apples to apples, head to the Configs tab.
- Review both models’ evaluation configurations, such as metrics, datasets, prompts, few-shot configs...
- If something looks off, it’s good to know before drawing conclusions! ✅

4/ Compare Predictions by Sample in the Details Tab 🔍
- Curious about how each model responds to specific inputs? The Details tab is your go-to!
- Select a Task (e.g., MuSR) and then a Subtask (e.g., Murder Mystery) and then press the Load Details button.
- Check out the side-by-side predictions and dive into the nuances of each model’s outputs.

5/ With this tool, it’s never been easier to explore how small changes between model versions affect performance on a wide range of tasks. Whether you’re a researcher or enthusiast, you can instantly visualize improvements and dive into detailed comparisons.

🚀 Try the 🤗 Open LLM Leaderboard Comparator now and take your model evaluations to the next level!


This has to be the first peak performance level use case of a non-autoregressive architecture for TTS. Flow matching for the win!!

Demo: https://huggingface.co/spaces/mrfakename/E2-F5-TTS
Model: https://huggingface.co/SWivid/E2-TTS


All the way from Korea, a novel approach called Mentor-KD significantly improves the reasoning abilities of small language models.

Mentor-KD introduces an intermediate-sized "mentor" model to augment training data and provide soft labels during knowledge distillation from large language models (LLMs) to smaller models.

Broadly, it’s a two-stage process:
1) Fine-tune the mentor on filtered Chain-of-Thought (CoT) annotations from an LLM teacher.
2) Use the mentor to generate additional CoT rationales and soft probability distributions.

The student model is then trained using:
- CoT rationales from both the teacher and mentor (rationale distillation).
- Soft labels from the mentor (soft label distillation).

Results show that Mentor-KD consistently outperforms baselines, with up to 5% accuracy gains on some tasks.

Mentor-KD is especially effective in low-resource scenarios, achieving comparable performance to baselines while using only 40% of the original training data.

This work opens up exciting possibilities for making smaller, more efficient language models better at complex reasoning tasks.

What are your thoughts on this approach?


https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Instruct-HF is now available in HuggingChat if you want to try it!

https://huggingface.co/chat/models/nvidia/Llama-3.1-Nemotron-70B-Instruct-HF

KingNish prithivMLmods jgitsolutions John6666 adorkin AtAndDev clem


🚨 I have $3,500 in Azure credits, including access to an H100 (96 Go), expiring on November 12, 2024.

I won’t be able to use it all myself, so I’m reaching out to the @huggingface community: Are there any open-source projets with data ready for some compute power?

 Let’s collaborate and make the most of it together 🔗


I am here to provide you with the premium codes you want just by informing me your requirements. #program #AI #code
Premium_Code
https://hf.co/chat/assistant/670fd7b2ad7fdbb38ff98102


Hi there HuggingFacers!

Have you ever dreamt of an improbable books crossover, like Frodo from 𝘓𝘰𝘳𝘥 𝘰𝘧 𝘵𝘩𝘦 𝘙𝘪𝘯𝘨𝘴 becoming the main character of the 𝘖𝘥𝘺𝘴𝘴𝘦𝘺 or Emma Bovary from 𝘔𝘢𝘥𝘢𝘮𝘦 𝘉𝘰𝘷𝘢𝘳𝘺 acting as a modern-days Shakespearean Juliet?

Well, all of this is now possible! I'm thrilled to introduce my latest opensource product for storytelling: 𝐛𝐨𝐨𝐤𝐬-𝐦𝐢𝐱𝐞𝐫-𝐚𝐢 𝐯𝟎.𝟎.𝟎 !

Built with ReactJS and shipped directly to you on Spaces thanks to Docker, this webapp combines the power of two AI tools:

- gpt-4o-mini by OpenAI, which takes care of cooking new and intriguing plots starting from the user's instructions, the titles and the summaries of the two books to mix (summaries are scraped through Wikipedia)
- text2img realtime API by ModelsLab, which provides a stable diffusion pipeline to create a thumbnail for your newly-generated story

Everything is provided under a simple and intuitive UI, which uses chatscope's React template kit.
Curious of trying? The app is already live at:

https://huggingface.co/spaces/as-cle-bert/books-mixer-ai

And you can also have a tour of the GitHub repo (and leave a little ⭐ while you're there):

https://github.com/AstraBert/books-mixer-ai

The documentation is still under construction, but will become available soon😊

Have fun!📚📚


Today I was able to solve a very difficult coding session with GPT-4o which ended up solving integrations on a very large scale.   So I decided to look a bit more into how its reasoners work.  Below is a fun markdown emoji outline about what I learned today and what I'm pursuing.

Hope you enjoy!  Cheers, Aaron.

Also here are my favorite last 4 spaces I am working on:
1. GPT4O:  https://huggingface.co/spaces/awacke1/GPT-4o-omni-text-audio-image-video
2. Claude:
 https://huggingface.co/spaces/awacke1/AnthropicClaude3.5Sonnet-ACW
3. MSGraph M365: https://huggingface.co/spaces/awacke1/MSGraphAPI
4. Azure Cosmos DB: Now with Research AI!  https://huggingface.co/spaces/awacke1/AzureCosmosDBUI

# 🚀 OpenAI's O1 Models: A Quantum Leap in AI

## 1. 🤔 From 🦜 to 🧠: O1's Evolution

- **Thinking AI**: O1 ponders before replying; GPT models just predict. 💡

## 2. 📚 AI Memory: 💾 + 🧩 = 🧠

- **Embeddings & Tokens**: Words ➡ vectors, building knowledge. 📖

## 3. 🔍 Swift Knowledge Retrieval

- **Vector Search & Indexing**: O1 finds info fast, citing reliable sources. 🔎📖

## 4. 🌳 Logic Trees with Mermaid Models

- **Flowchart Reasoning**: O1 structures thoughts like diagrams. 🎨🌐

## 5. 💻 Coding Mastery

- **Multilingual & Current**: Speaks many code languages, always up-to-date. 💻🔄

## 6. 🏆 Breaking Records

- **92.3% MMLU Score**: O1 outperforms humans, setting new AI standards. 🏅

## 7. 💡 Versatile Applications

- **Ultimate Assistant**: From fixing code to advancing research. 🛠🔬

## 8. 🏁 Racing Toward AGI

- **OpenAI Leads**: O1 brings us closer to true AI intelligence. 🚀

## 9. 🤖 O1's Reasoning Pillars

- **🧠 Chain of Thought**: Step-by-step logic.
- **🎲 MCTS**: Simulates options, picks best path.
- **🔍 Reflection**: Self-improves autonomously.
- **🏋♂ Reinforcement Learning**: Gets smarter over time.

---

*Stay curious, keep coding!* 🚀


In regards to the latest mistral model and GGUFs for it:

Yes, they may be subpar and may require changes to llama.cpp to support the interleaved sliding window

Yes, I got excited when a conversion worked and released them ASAP

That said, generation seems to work right now and seems to mimic the output from spaces that are running the original model

I have appended -TEST to the model names in an attempt to indicate that they are not final or perfect, but if people still feel mislead and that it's not the right thing to do, please post (civilly) below your thoughts, I will highly consider pulling the conversions if that's what people think is best. After all, that's what I'm here for, in service to you all !


🔥 Meta AI just blessed us with CoTracker v3, bleeding edge point tracking foundation model 🤩
model: https://huggingface.co/facebook/cotracker3
demo: https://huggingface.co/spaces/facebook/cotracker

John6666 jgitsolutions KvrParaskevi Rostenbach LeonceNsh


Terribly excited about open-source + on-device AI these days! Great to see @qualcomm release 80+ models optimized and curated for their devices and chips on HF: https://huggingface.co/qualcomm

DmitryRyumin vladbogo harpreetsahota adamelliotfields banghua mvaloatto lunarflu osanseviero marceloxp ajibawa-2023 terpjwu1 danielhanchen yjernite samusenps realtamimuddin

lunarflu osanseviero danielus ehristoforu siyengfeng yjernite realtamimuddin


Want more “good machine learning” in your X feed? Here is a new Space for you:
🔔 Top HF Users To Follow On X - https://huggingface.co/spaces/mvaloatto/HF2X

Ever since I fell down the AI rabbit hole, it hasn’t been super easy to spot and follow the most impactful Hugging Face contributors on X. So, inspired by @Weyaxi leaderboards, I decided to create a list just for this purpose.

Why, you ask?

First, it’s quite surprising how so many talented AI pioneers and independent contributors on X don't get the visibility/reach you might expect. Sad but true: follower count doesn't always match up with the value or innovation an individual brings to the table (just stating the obvious here).

Open source AI, in particular, thrives not just on innovation but also on the collective spirit of its believers and builders. With Hugging Face standing out as a prime hub for top AI engineers and contributors, compiling a directory of X profiles from influential figures on this platform felt like a natural step.

This Space aims to not only connect these top contributors but also guide open AI enthusiasts and newcomers towards the field's leading lights.

I put this modest page together using some web scraping and what I remember from my web dev class ages ago! Suggestions/likes are welcome - I’m hoping to keep tweaking/upgrading it, especially if you all find it useful.

Now, let’s follow each other! It’s time to accelerate the dissemination of our ideas, encourage collaboration within our community, and ensure that open AI developments receive the attention and recognition they deserve. 🔥

clem DmitryRyumin osanseviero Alexandro14 jeffboudier lunarflu alielfilali01 Weyaxi nickandbro danielhanchen mattmdjaga fffiloni silveriomf pcuenq carlosmoises

zaghadon marceloxp kramp Weyaxi terpjwu1 carlosmoises mathiasn1


@jinaai, we've recently launched an interesting model: https://huggingface.co/jinaai/jina-colbert-v1-en. In this post, I'd like to give you a quick introduction to ColBERT: the multi-vector search & late interaction retriever.

As you may already know, we've been developing embedding models such as https://huggingface.co/jinaai/jina-embeddings-v2-base-en for some time. These models, often called 'dense retrievers', generate a single representation for each document.

Embedding models like Jina-v2 have the advantage of quick integration with vector databases and good performance within a specific domain.

When discussing tasks within a specific domain, it means embedding models can perform very well by "seeing similar distributions". However, this also suggests that they might only perform "okay" on tasks outside of that domain and require fine-tuning.

Now, let's delve into multi-vector search and late-interaction models. The idea is quite simple:

1. During model training, you apply dimensionality reduction to decrease the vector dimensionality from 768 to 128 to save storage.
2. Now, with one query and one document, you match each query token embedding against every token embedding in the document to find the maximum similarity score. Repeat this process for each token in the query, from the second to the last, and then sum up all the maximum similarity scores.

This process is called multi-vector search because if your query has 5 tokens, you're keeping 5 * 128 token embeddings. The "max similarity sum-up" procedure is termed late interaction.

Multi-vector & Late interaction retrievers have the advantage of:

1. Excellent performance outside of a specific domain since they match at a token-level granularity.
2. Explainability: you can interpret your token-level matching and understand why the score is higher/lower.

Try our first multi-vector search at https://huggingface.co/jinaai/jina-colbert-v1-en and share your feedback!

clem osanseviero taylorbollman dev7halo victor dillfrescott nuggi sbrandeis


https://qwen.readthedocs.io/ 🔥 The official doc of Qwen1.5 is coming! This is a bilingual doc (English and Chinese, and it will be multilingual if I have time for them). The doc includes instructions for simple inference, running locally with GGUF, ollama, etc., quantization, finetuning, deployment, etc. We will continue adding more stuff to the doc. Stay tuned!


🚀🧙🏼♂Introducing OpenHermesPreferences: the largest open AI feedback dataset for RLHF & DPO

> Using LLMs to improve other LLMs, at scale!

Built in collaboration with the H4 Hugging Face team, it's a 1M preferences dataset on top of the amazing @teknium 's dataset.

Dataset:
https://huggingface.co/datasets/argilla/OpenHermesPreferences

The dataset is another example of open collaboration:

> The H4 team created responses with Mixtral using llm-swarm

> Argilla created responses with NousResearch Hermes-2-Yi-34B using distilabel

> The H4 ranked these responses + original response with PairRM from AllenAI, University of Southern California, Zhejiang University (@yuchenlin @DongfuTingle and colleagues)

We hope this dataset will help the community's research efforts towards understanding the role of AI feedback for LLM alignment.

We're particularly excited about the ability of filtering specific subsets to improve LLM skills like math or reasoning.

Here's how easy it is to filter by subset:

```
ds = load_dataset("HuggingFaceH4/OpenHermesPreferences", split="train")

# Get the categories of the source dataset
# ['airoboros2.2', 'CamelAI', 'caseus_custom', ...]
sources = ds.unique("source")

# Filter for a subset
ds_filtered = ds.filter(lambda x : x["source"] in ["metamath", "EvolInstruct_70k"], num_proc=6)
```

As usual, all the scripts to reproduce this work are available and open to the community!

https://huggingface.co/datasets/argilla/OpenHermesPreferences/tree/main

So fun collab between @vwxyzjn , @plaguss, @kashif, @philschmid &  @lewtun!

Open Source AI FTW!

davanstrien plaguss philschmid osanseviero alielfilali01 Rybens lewtun xiaotianhan gabrielmbmb clem vladbogo koutch kramp thivav Nindaleth

philschmid osanseviero alielfilali01 clem


MobileLLM: Optimizing Sub-billion Parameter Language Models for
  On-Device Use Cases (2402.14905)

MobileLLM

Optimizing Sub-billion Parameter Language Models for On-Device Use Cases

https://huggingface.co/papers/2402.14905

paper addresses the growing need for efficient large language models (LLMs) on mobile devices, driven by increasing cloud costs and latency concerns. We focus on designing top-quality LLMs with fewer than a billion parameters, a practical choice for mobile deployment. Contrary to prevailing belief emphasizing the pivotal role of data and parameter quantity in determining model quality, our investigation underscores the significance of model architecture for sub-billion scale LLMs. Leveraging deep and thin architectures, coupled with embedding sharing and grouped-query attention mechanisms, we establish a strong baseline network denoted as MobileLLM, which attains a remarkable 2.7%/4.3% accuracy boost over preceding 125M/350M state-of-the-art models. Additionally, we propose an immediate block-wise weight sharing approach with no increase in model size and only marginal latency overhead. The resultant models, denoted as MobileLLM-LS, demonstrate a further accuracy enhancement of 0.7%/0.8% than MobileLLM 125M/350M. Moreover, MobileLLM model family shows significant improvements compared to previous sub-billion models on chat benchmarks, and demonstrates close correctness to LLaMA-v2 7B in API calling tasks, highlighting the capability of small models for common on-device use cases.


Exciting news for `bitsandbytes`! We're thrilled to announce the release of the initial version of our new documentation: https://huggingface.co/docs/bitsandbytes/main/en/index .

Please let us know what you think: Your feedback is essential to us, and we would greatly appreciate any insights you have on how we can further enhance it or even better be happy to merge your contributions, filling in some blanks: Especially doc-strings are still a big topic and there several placeholder that would be super helpful to have filled in. Please post your feedback here: https://github.com/TimDettmers/bitsandbytes/discussions/1090

Since taking over maintenance together with Younes Belkada and since Hugging Face graciously agreed to support the library, we've already made enormous strides and community contributions have sprung back to life: It's so motivating to have so many knowledgeable contributors that often invest extensive free-time and bring their unique ideas to the table.

A notable example are our ongoing efforts to enable cross-platform support, including Intel, Apple Silicon, AMD, and Windows. Simultaneously, we're working diligently to streamline community contributions in BNB, making the process more accessible for everyone. A heartfelt thank you to all who have contributed thus far!

With HuggingFace's committed to supporting bitsandbytes going forward, we're sure to promptly respond to and integrate additional community contributions.

Looking forward to growing bitsandbytes further as part of the FOSS community: pushing forward the state of the art in democratization of AI!

Titus-von-Koeller clem osanseviero gsarti ybelkada mdouglas


First big community contribution on our evaluation suite, lighteval ⛅

@Ali-C137  added 3 evaluation tasks in Arabic:
- ACVA, a benchmark about Arabic culture
- MMLU, translated
- Exams, translated
(datasets provided/translated by the AceGPT team)

Congrats to them!
https://github.com/huggingface/lighteval/pull/44

clem alielfilali01 macadeliccc osanseviero vladbogo


"The principle of explainability of ai and its application in organizations"
Louis Vuarin, Véronique Steyer
—› 📔 https://doi.org/10.3917/res.240.0179

ABSTRACT: The explainability of Artificial Intelligence (AI) is cited in the literature as a pillar of AI ethics, yet few studies explore its organizational reality. This study proposes to remedy this shortcoming, based on interviews with actors in charge of designing and implementing AI in 17 organizations. Our results highlight: the massive substitution of explainability by the emphasis on performance indicators; the substitution of the requirement of understanding by a requirement of accountability; and the ambiguous place of industry experts within design processes, where they are employed to validate the apparent coherence of ‘black-box’ algorithms rather than to open and understand them. In organizational practice, explainability thus appears sufficiently undefined to reconcile contradictory injunctions. Comparing prescriptions in the literature and practices in the field, we discuss the risk of crystallizing these organizational issues via the standardization of management tools used as part of (or instead of) AI explainability.

Vuarin, Louis, et Véronique Steyer. « Le principe d’explicabilité de l’IA et son application dans les organisations », Réseaux, vol. 240, no. 4, 2023, pp. 179-210.

#ArtificialIntelligence #AIEthics #Explainability #Accountability

gsarti osanseviero Apparro Dlbk Alexandro14 clem Concor bukaka OmbelineM


Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity
  Tracking (2402.14811)

💥 Today in Interpretability & Analysis of LMs: Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking by @nikhil07prakash @tamarott @TalHaklay @belinkov @davidbau

Fine-tuning is commonly used to improve LM’s capabilities, but its impact on model-internal mechanisms remains poorly understood.

This work evaluates the impact of fine-tuning from a mechanistic perspective, using entity tracking in fine-tuned LLaMA 7B variants as a test bench.

Authors use path patching to highlight how fine-tuned models largely employ the same circuits as their pre-trained counterparts to solve entity tracking. Desiderata-based Component Masking (DCM) is used to discern the function of circuit components, finding that even the functionality of the circuit components remains consistent after fine-tuning.

Where do the gains stem from, then? Using Cross-Model Activation Patching (CMAP), authors show the benefits of fine-tuning are largely derived from an improved ability of circuit components to encode important task-relevant information rather than an overall functional rearrangement. Interestingly, fine-tuned activations are compatible with the base model despite no explicit constraint during representation learning.

🌐 Website: http://finetuning.baulab.info/
🤖 Model: https://huggingface.co/nikhil07prakash/float-7b
📄 Paper: https://huggingface.co/papers/2402.14811

🔍 All daily picks: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9


Genie

Generative Interactive Environments

https://huggingface.co/papers/2402.15391

introduce Genie, the first generative interactive environment trained in an unsupervised manner from unlabelled Internet videos. The model can be prompted to generate an endless variety of action-controllable virtual worlds described through text, synthetic images, photographs, and even sketches. At 11B parameters, Genie can be considered a foundation world model. It is comprised of a spatiotemporal video tokenizer, an autoregressive dynamics model, and a simple and scalable latent action model. Genie enables users to act in the generated environments on a frame-by-frame basis despite training without any ground-truth action labels or other domain-specific requirements typically found in the world model literature. Further the resulting learned latent action space facilitates training agents to imitate behaviors from unseen videos, opening the path for training generalist agents of the future.


A Closer Look at the Limitations of Instruction Tuning (2402.05119)

"A Closer Look at the Limitations of Instruction Tuning" is a new paper that explores the efficacy and limitations of Instruction Tuning (IT) in Large Language Models (LLMs) for conversational agents. The authors conduct a series of experiments using both LoRA fine-tuning (LFT) and standard full-parameter fine-tuning (SFT) across various LLMs and IT datasets.

The key findings are:
* LoRA fine-tuning (LFT) preserves the pre-training token distribution while SFT doesn't. This indicates that using LFT, post fine-tuning the model still heavily relies on the pre-training and doesn't acquire new information.
* Dataset scaling is ineffective for LFT - experiments show that scaling the dataset size 52x or even 326x doesn't improve the performance.
* LoRA fine-tuning mainly enhances response initiation and style without substantial knowledge enhancement.
* Full-parameter fine-tuning tends to degrade LLM knowledge base and increase hallucination occurrences.
* Popular other methods and adjustments fail to significantly outperform simple LoRA fine-tuned models in terms of conversational quality and accuracy.

Congrats to the authors @Sreyan88 and others for their work!

Paper: https://huggingface.co/papers/2402.05119

Hopkinx osanseviero ioanacroi kaleinaNyan clem Felladrin RamaniD sonalkum TheDetective Sreyan88 Arjun4707 TerraNull schoolta


🌟✨ Exciting Announcement: NVIDIA AI Foundation Models ✨🌟

🚀 Interact effortlessly with the latest SOTA AI model APIs, all optimized on the powerful NVIDIA accelerated computing stack-right from your browser! 💻⚡

🔗 Web Page: https://catalog.ngc.nvidia.com/ai-foundation-models

🌟🎯 Favorites:

🔹 Code Generation:
1⃣ Code Llama 70B 📝🔥: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/codellama-70b
Model 🤖: https://huggingface.co/codellama/CodeLlama-70b-hf

🔹 Text and Code Generation:
1⃣ Gemma 7B 💬💻: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/gemma-7b
Model 🤖: https://huggingface.co/google/gemma-7b
2⃣ Yi-34B 📚💡: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/yi-34b
Model 🤖: https://huggingface.co/01-ai/Yi-34B

🔹 Text Generation:
1⃣ Mamba-Chat 💬🐍: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/mamba-chat
Model 🤖: https://huggingface.co/havenhq/mamba-chat
2⃣ Llama 2 70B 📝🦙: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/llama2-70b
Model 🤖: https://huggingface.co/meta-llama/Llama-2-70b

🔹 Text-To-Text Translation:
1⃣ SeamlessM4T V2 🌐🔄: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/seamless-m4t2-t2tt
Model 🤖: https://huggingface.co/facebook/seamless-m4t-v2-large

🔹 Image Generation:
1⃣ Stable Diffusion XL 🎨🔍: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/sdxl

🔹 Image Conversation:
1⃣ NeVA-22B 🗨📸: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/neva-22b

🔹 Image Classification and Object Detection:
1⃣ CLIP 🖼🔍: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/clip

🔹 Voice Conversion:
1⃣ Maxine Voice Font 🗣🎶: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/voice-font

🔹 Multimodal LLM (MLLM):
1⃣ Kosmos-2 🌐👁: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/kosmos-2

MiSTe-R vladbogo ssbagpcm osanseviero samusenps Josemalyson eramax s3nh Alexandro14 Csplk ArunkumarVR xiaotianhan Samoed shubhamg2208 Antreas dillfrescott yahma alielfilali01 Nymbo


LLM Agents can Autonomously Hack Websites (2402.06664)

"LLM Agents can Autonomously Hack Websites" is a new paper that investigates the capacity of LLMs to autonomously execute cybersecurity attacks on websites, such as SQL injections without human guidance.

Key points:
* It uses a LLM integrated with Playwright, a headless web browser, enabling automated web interactions through function calling.
* It gives access to the LLM to 7 web hacking documents and planning capabilities through specific prompting, without disclosing the exact methods to prevent misuse.

GPT-4 achieves a 73.3% success rate on the tested vulnerabilities, emphasizing the potential cybersecurity risks posed by advanced LLMs. Other open models cannot yet perform these types of attacks (results in screenshot).

Congrats to the authors for their work!

Paper: https://huggingface.co/papers/2402.06664

Kukedlc alwint3r osanseviero leeloolee theainerd WhenceFade


VLMs have a resolution problem, which prevents them from finding small details in large images. In my community blog post, I discuss the ways to solve it and describe the details of MC-LLaVA architecture - https://huggingface.co/blog/visheratin/vlm-resolution-curse

Check it out, and let me know what you think!

osanseviero samusenps georgechkhvirkia clem radames duxx Alexandro14 Molbap oldsea


Today, I’m thrilled to release a project I’ve been working on for the past couple weeks in collaboration with Hugging Face: the TTS Arena.

The TTS Arena, inspired by LMSys's Chatbot Arena, allows you to enter text which will be synthesized by two SOTA models. You can then vote on which model generated a better sample. The results will be published on a publicly-accessible leaderboard.

We’ve added several open access models, including Pheme, MetaVoice, XTTS, OpenVoice, & WhisperSpeech. It also includes the proprietary ElevenLabs model.

If you have any questions, suggestions, or feedback, please don’t hesitate to DM me on X (https://twitter.com/realmrfakename) or open a discussion in the Space. More details coming soon!

Try it out: https://huggingface.co/spaces/TTS-AGI/TTS-Arena

tomaarsen Korakoe mlabonne samusenps clem kurianbenoy reach-vb Alexandro14 Wauplin realsammyt mvaloatto julien-c ameerazam08 clefourrier alielfilali01 Suparious michaljunczyk Nymbo

rafaelpierrehf vladbogo Korakoe clem reach-vb mathiasn1 ameerazam08 Nymbo


Check out the demo of SPIN-Diffusion made by @angelahzyuan at: https://huggingface.co/spaces/UCLA-AGI/SPIN-Diffusion-demo-v1

osanseviero angelahzyuan MaziyarPanahi samusenps akhaliq mohammed-17


VideoPrism: A Foundational Visual Encoder for Video Understanding (2402.13217)

VideoPrism is a new video encoder that improves video understanding through a unique training strategy, using a vast dataset (36 million high-quality video-caption pairs and 582 million video clips) for comprehensive learning.

Key points:
* It employs a two-stage training approach, initially aligning video and text encoders, followed by an enhanced video-only masked autoencoding process to learn appearance and motion.
* It achieves superior performance in a wide array of tasks, such as general video understanding, zero-shot video-text retrieval, video captioning, QA, and computer vision for science, having top performance on 30 out of 33 benchmarks.

Congrats to the authors for their work!

Paper: https://huggingface.co/papers/2402.13217

Kukedlc guitarliu0320 osanseviero clem liusuchao Alexandro14 clefourrier


OpenCodeInterpreter: Integrating Code Generation with Execution and
  Refinement (2402.14658)

OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement

https://huggingface.co/papers/2402.14658

The introduction of large language models has significantly advanced code generation. However, open-source models often lack the execution capabilities and iterative refinement of advanced systems like the GPT-4 Code Interpreter. To address this, we introduce OpenCodeInterpreter, a family of open-source code systems designed for generating, executing, and iteratively refining code. Supported by Code-Feedback, a dataset featuring 68K multi-turn interactions, OpenCodeInterpreter integrates execution and human feedback for dynamic code refinement. Our comprehensive evaluation of OpenCodeInterpreter across key benchmarks such as HumanEval, MBPP, and their enhanced versions from EvalPlus reveals its exceptional performance. Notably, OpenCodeInterpreter-33B achieves an accuracy of 83.2 (76.4) on the average (and plus versions) of HumanEval and MBPP, closely rivaling GPT-4's 84.2 (76.2) and further elevates to 91.6 (84.6) with synthesized human feedback from GPT-4. OpenCodeInterpreter brings the gap between open-source code generation models and proprietary systems like GPT-4 Code Interpreter.

Jenish-23 mrfakename radames Shawnlight Kukedlc vladbogo DmitryRyumin boapps nirajandhakal


🤗 Sentence Transformers v2.4.0 for embedding models is now out! It introduces a lot of powerful features, such as:

1. Matryoshka Loss function - you can now train & perform inference on 🪆 Matryoshka Embedding models. See also our blogpost: https://huggingface.co/blog/matryoshka

2. CoSENTLoss & AnglELoss: State of the art loss functions. These are quite interesting, they outperform CosineSimilarityLoss on nearly all benchmarks as a drop-in replacement! See also the docs: https://sbert.net/docs/package_reference/losses.html#cosentloss

3. Prompt templates: Many popular models such as https://huggingface.co/intfloat/multilingual-e5-large and https://huggingface.co/BAAI/bge-large-en-v1.5 prefix their texts with prompts, so this adds configuration options to automatically include prompts using `model.encode(..., prompt_name="query")` which will include a prompt with the name "query". More info in the docs: https://sbert.net/examples/applications/computing-embeddings/README.html#prompt-templates

4. Instructor support: Support for the INSTRUCTOR line of models, such as https://huggingface.co/hkunlp/instructor-large. Learn how to use them here: https://sbert.net/docs/pretrained_models.html#instructor-models

5. Removed NLTK & sentencepiece dependencies: Should allow for a smaller installation & a slightly faster import!

6. Updated documentation: a new Loss Overview section: https://sbert.net/docs/training/loss_overview.html and more detailed loss functions: https://sbert.net/docs/package_reference/losses.html

And much more! See the full release notes here: https://github.com/UKPLab/sentence-transformers/releases/tag/v2.4.0

Some more very exciting updates are still on the horizon!

radames Azamat1k Kukedlc eramax victor alvarobartt asoria bfn DeepMount00


Just when I was about to go to bed....... Here we go again

osanseviero lysandre reach-vb mathiasn1 pcuenq jpm8998 riddhiman victor julien-c radames vladbogo Tom-Neverwinter Azamat1k rafaelpierrehf himasai9711 burtugeey sbrandeis

regisss ybelkada osanseviero reach-vb riddhiman clem julien-c radames


Compared Stable Diffusion 3 with Dall-E3 and results are mind blowing.

SD 3 can follow prompts many times better than SD 1.5 or SDXL. It is even better than Dall-E3 in following text / spelling prompts.

The realism of the SD 3 can't be even compared with Dall-E3, since every Dall-E3 output is like a digital render.

Can't wait to get approved of Stability AI early preview program to do more intensive testing.

Some people says be skeptical about cherry picking. I agree but I hope that these Stability AI released images are not that heavy cherry picking.

You can see SD3 vs Dall-E3 comparison here : https://youtu.be/DJxodszsERo

Dlbk mdzihad vladbogo Kukedlc admarcosai danielus Rsln SANCHEZRICK KADS-dev Glavin001 NHLOCAL


YOLOv9: Learning What You Want to Learn Using Programmable Gradient
  Information (2402.13616)

🎉✨ Exciting Research Alert! YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information 🚀

YOLOv9 is the latest breakthrough in object detection!

📄 Title: YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information

👥 Authors: Chien-Yao Wang et al.
📅 Published: ArXiv, February 2024

🔗 Paper: https://huggingface.co/papers/2402.13616
🔗 Model 🤖: https://huggingface.co/adonaivera/yolov9
🔗 Repo: https://github.com/WongKinYiu/yolov9

🚀 Don't miss out on this cutting-edge research! Explore YOLOv9 today and stay ahead of the curve in the dynamic world of computer vision. 🌟

🔍 Keywords: #YOLOv9 #ObjectDetection #DeepLearning #ComputerVision #Innovation #Research #ArtificialIntelligence

Kukedlc samusenps osanseviero taufiqdp Dlbk mathiasn1 kadirnar vladbogo dillfrescott jairo

samusenps osanseviero clem Dlbk kadirnar dillfrescott femboysLover ntdas


Hugging Face announces Cosmo 1B, a fully open sourced Phi competitor with an open sourced dataset. The dataset references various articles and textbooks as "seed data" to generate conversations. Licensed under the Apache 2.0 license. The dataset, dubbed "Cosmopedia," is published on the Hugging Face Hub under the Apache 2.0 license. It was generated using Mixtral 8x7B with various sources (AutoMathText, OpenStax, WikiHow, etc) as "seed data."

Model: https://huggingface.co/HuggingFaceTB/cosmo-1b
Dataset: https://huggingface.co/datasets/HuggingFaceTB/cosmopedia

Kukedlc wesley-mtk osanseviero clem kramp Dlbk radames admarcosai compressionsavant Nymbo

samusenps osanseviero clem Dlbk s3nh radames MaziyarPanahi


Enhanced Hallucination Detection in Neural Machine Translation through
  Simple Detector Aggregation (2402.13331)

🔍 Today's pick in Interpretability & Analysis of LMs: Enhanced Hallucination Detection in Neural Machine Translation through Simple Detector Aggregation by A. Himmi G. Staerman M. Picot @Colombo @nunonmg

Previous work on hallucination detection for MT showed that different detectors excel at detecting different types of hallucinations.

In this context, detectors based solely on model internals such as input contributions or sequence log-probabilities fare well on fully-detached hallucinations but show limited performances on oscillatory hallucinations, where ad-hoc trained detectors are still the best-performing methods.

This work proposes Simple Detectors Aggregation (STARE), an aggregation procedure to leverage detectors’ complementary strengths. Authors experiment with two popular hallucination detection benchmarks (LFAN-HALL and HalOmi), showing that STARE outperforms single detectors and other aggregation baselines.

Results obtained aggregating internal detectors highlight how model-based features that are readily available as generation byproducts can outperform computationally expensive ad-hoc solutions.

📄 Paper: https://huggingface.co/papers/2402.13331

🔍 All daily picks in LM interpretability: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9

samusenps lunarflu osanseviero Kukedlc nunonmg Alexandro14


LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (2402.13753)

LongRoPE

Extending LLM Context Window Beyond 2 Million Tokens

https://huggingface.co/papers/2402.13753

Large context window is a desirable feature in large language models (LLMs). However, due to high fine-tuning costs, scarcity of long texts, and catastrophic values introduced by new token positions, current extended context windows are limited to around 128k tokens. This paper introduces LongRoPE that, for the first time, extends the context window of pre-trained LLMs to an impressive 2048k tokens, with up to only 1k fine-tuning steps at within 256k training lengths, while maintaining performance at the original short context window. This is achieved by three key innovations: (i) we identify and exploit two forms of non-uniformities in positional interpolation through an efficient search, providing a better initialization for fine-tuning and enabling an 8x extension in non-fine-tuning scenarios; (ii) we introduce a progressive extension strategy that first fine-tunes a 256k length LLM and then conducts a second positional interpolation on the fine-tuned extended LLM to achieve a 2048k context window; (iii) we readjust LongRoPE on 8k length to recover the short context window performance. Extensive experiments on LLaMA2 and Mistral across various tasks demonstrate the effectiveness of our method. Models extended via LongRoPE retain the original architecture with minor modifications to the positional embedding, and can reuse most pre-existing optimizations.

eramax Madisen vladbogo PrathmeshZ clem NickyNicky lunarflu osanseviero thughost Ji-Ha admarcosai PedramR xiaotianhan

tomaarsen FranZuzz omaryshchenko clem lunarflu osanseviero Ji-Ha charlieCs


Data map of the languages of https://huggingface.co/datasets/CohereForAI/aya_dataset

osanseviero clem TuringsSolutions samusenps lunarflu kurianbenoy sayhan not-lain lilouuch AliArshad shivi OmbelineM


Rephrasing the Web: A Recipe for Compute and Data-Efficient Language
  Modeling (2401.16380)

Web Rephrase Augmented Pre-training (WRAP) enhances language model training efficiency by transforming documents into structured formats.

Key aspects:
* Utilizes an instruction-tuned model to rephrase web content into styles such as Wikipedia or Q/A, creating a blend of synthetic and real data for training.
* Demonstrated improvements of over 10% better perplexity, alongside more than 2% increase in zero-shot question-answering accuracy.

Congrats to the authors for their work!

Paper: https://huggingface.co/papers/2401.16380

osanseviero clem lunarflu BroCodeR Alexandro14 kotyKD abdullah


📚🔎 If you're building RAG applications, you should check this out:

⚙ I've built a new space to let you visualize the chunks you get with different text splitting methods!

➡ Visualize your chunks here:
https://huggingface.co/spaces/m-ric/chunk_visualizer

MoritzLaurer vladbogo DmitryRyumin SolarWindRider andrewrreed aari1995 osanseviero clem lunarflu sds20 sparkonelabs mushimeraki


looking at the tokenizer and the naming (“_en“), Google Gemma is very likely to have a multilingual counterpart. 👀

Thoughts?

vladbogo sayhan osanseviero clem noobmldude


I am thrilled to announce Gemma, new 2B and 7B models from Google, based on the same research and technology used to train the Gemini models! These models achieve state-of-the-art performance for their size, and are launched across Transformers, Google Cloud, and many other surfaces worldwide starting today.

Get started using and adapting Gemma in the model Collection: https://huggingface.co/collections/google/gemma-release-65d5efbccdbb8c4202ec078b

These launches are the product of an outstanding collaboration between the Google DeepMind and Hugging Face teams over the last few months -- very proud of the work both teams have done, from integration with Vertex AI to optimization across the stack. Read more about the partnership in the main launch by @philschmid @osanseviero @pcuenq on the launch blog: https://huggingface.co/blog/gemma

More information below if you are curious about training details, eval results, and safety characteristics!

Gemma Tech Report: https://goo.gle/GemmaReport
Launch announcement: https://blog.google/technology/developers/gemma-open-models/

algoartist megrisdal andysalerno osanseviero victor BrigitteTousi alvarobartt abidlabs clem jeffboudier joaogante DmitryRyumin manvinder kramp VictorSanh Alexandro14 samforeman vladbogo taufiqdp pcuenq Kukedlc Madisen chansung MoritzLaurer m-ric andrewrreed Anthonyg5005 abhishek mathiasn1 nicolasdec Cstark ethix

alvarobartt abidlabs clem thibaultM osanseviero AP123 jeffboudier joaogante VictorSanh Alexandro14 pcuenq Kukedlc Madisen Bils chansung SanRosenberg andrewrreed Bolamide ZenithBrew allknowingroger abhishek sugatoray

clem osanseviero VictorSanh Kukedlc Madisen chansung abhishek

osanseviero Kukedlc Madisen abhishek Cstark


LLM hallucination detection papers be like:

* check the image to get the joke 👀

fahnub Kukedlc Doma02 gsarti sayhan MarinaraSpaghetti vijay8642 dashfunnydashdash Vermilion3154 Kirkalish notoookay


 Neural Network Diffusion

https://huggingface.co/papers/2402.13144

Diffusion models have achieved remarkable success in image and video generation. In this work, we demonstrate that diffusion models can also generate high-performing neural network parameters. Our approach is simple, utilizing an autoencoder and a standard latent diffusion model. The autoencoder extracts latent representations of a subset of the trained network parameters. A diffusion model is then trained to synthesize these latent parameter representations from random noise. It then generates new representations that are passed through the autoencoder's decoder, whose outputs are ready to use as new subsets of network parameters. Across various architectures and datasets, our diffusion process consistently generates models of comparable or improved performance over trained networks, with minimal additional cost. Notably, we empirically find that the generated models perform differently with the trained networks. Our results encourage more exploration on the versatile use of diffusion models.

osanseviero kgourgou SanRosenberg vladbogo gsarti Salwa-Zeitoun SuperShark takeraparterer DataSoul


New base pretrained models on the Open LLM Leaderboard!

Two new OSS models by Google, who's getting back in the game 😎
The 7B is 2nd of the leaderboard, and better than Mistral (notably on GSM8K, aka math).

https://huggingface.co/google/gemma-7b
https://huggingface.co/google/gemma-2b

Check more results on the leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard

tomaarsen Citaman Sylvestre beomi eramax manvinder macaulj1 osanseviero lazarustda Alexandro14 kramp Rayabarapu clem trisfromgoogle VictorSanh vladbogo mrfakename Kukedlc johndpope alielfilali01 BrigitteTousi Rybens


Exciting news for Indic LLMs! 🚀

Sarvam AI just released high-quality, curated dataset with multi-turn conversations in English, Hindi, and Hinglish! 💎 With a whopping 100K samples! 🤯
Check it out: https://huggingface.co/datasets/sarvamai/samvaad-hi-v1

Who's going to finetune high-quality SFT models on this dataset? ✨
if you are interested in pushing the boundaries with respect to Indic LLMs, join the discord channel: https://discord.gg/hugging-face-879548962464493619

osanseviero rajveer43 Rayabarapu clem masterfury alielfilali01