Self-Play Fine-Tuning Converts Weak Language Models to Strong Language
  Models (2401.01335)

🔥 Community and Data Quality Are More For Alignment

A recipe to replicate SPIN (Self-Play Fine Tuning) with 30x less data:

🗣 50K samples vs 1.8K prompts curated by the 350+ amazing DIBT contributors.
⚗ Distillation of Mistral Large instead of OpenAI
🙌 Open data & code with ⚗distilabel

SPIN Paper:
https://huggingface.co/papers/2401.01335

SPIN DIBT Collection with datasets and models:
https://huggingface.co/collections/argilla/dibt-prompt-collective-spin-65ef59062518776024395fc3

Repo:
https://github.com/argilla-io/distilabel-spin-dibt

Joint work with the amazing DIBT community 👇
@aashish1904, @flozi00, @sayhan, @munish0838, @0-hero, @dvilasuero, @eren23, @davanstrien, @ahnz, @BlackKakapo, @kitano-o, @mmhamdy, @sdiazlor, @Stopwolf, @gabrielmbmb, @tculler91, @plaguss, @ignacioct, @Hugi-R, @davidberenstein1957, @Korla, @alvarobartt, @Hugs4Llamas, @Sumandora, @nataliaElv, @jfcalvo, @Averill, @steventrouble, @vasilis, @aeros93, @kayyshf, @thomasgauthier, @jeromebas, @Ameeeee, @ayoubelmhamdi, @TuringsSolutions, @efels, @Haleyok, @abrazador, @emessy, @Nindaleth, @burtenshaw, @vicgalle, @CortexPE, @casey-martin, @Leire-aguirre-eguiluz, @mrfakename, @Portias600kNeurons, @nathaliepett, @Filippo

davanstrien nataliaElv ignacioct victor davidberenstein1957 munish0838 0-hero dvilasuero julien-c pacoid plaguss osanseviero mmhamdy alvarobartt gabrielmbmb mihaylovnikitos TuringsSolutions clem hughte BlackKakapo

davanstrien nataliaElv flozi00 davidberenstein1957 julien-c pacoid plaguss giux78 osanseviero mmhamdy gabrielmbmb BlackKakapo abdullahalzubaer clem

davanstrien davidberenstein1957 julien-c pacoid osanseviero mmhamdy gabrielmbmb BlackKakapo clem thomasgauthier

davidberenstein1957 julien-c pacoid osanseviero mmhamdy BlackKakapo clem


MoAI: Mixture of All Intelligence for Large Language and Vision Models (2403.07508)

MoAI

Mixture of All Intelligence for Large Language and Vision Models

https://huggingface.co/papers/2403.07508

The rise of large language models (LLMs) and instruction tuning has led to the current trend of instruction-tuned large language and vision models (LLVMs). This trend involves either meticulously curating numerous instruction tuning datasets tailored to specific objectives or enlarging LLVMs to manage vast amounts of vision language (VL) data. However, current LLVMs have disregarded the detailed and comprehensive real-world scene understanding available from specialized computer vision (CV) models in visual perception tasks such as segmentation, detection, scene graph generation (SGG), and optical character recognition (OCR). Instead, the existing LLVMs rely mainly on the large capacity and emergent capabilities of their LLM backbones. Therefore, we present a new LLVM, Mixture of All Intelligence (MoAI), which leverages auxiliary visual information obtained from the outputs of external segmentation, detection, SGG, and OCR models. MoAI operates through two newly introduced modules: MoAI-Compressor and MoAI-Mixer. After verbalizing the outputs of the external CV models, the MoAI-Compressor aligns and condenses them to efficiently use relevant auxiliary visual information for VL tasks. MoAI-Mixer then blends three types of intelligence (1) visual features, (2) auxiliary features from the external CV models, and (3) language features by utilizing the concept of Mixture of Experts. Through this integration, MoAI significantly outperforms both open-source and closed-source LLVMs in numerous zero-shot VL tasks, particularly those related to real-world scene understanding such as object existence, positions, relations, and OCR without enlarging the model size or curating extra visual instruction tuning datasets.

sugatoray jucamohedano maywell damerajee


Working on a new open-source project to showcase https://huggingface.co/levihsu/OOTDiffusion  🤗

But it's not a Space 🫢  Can you guess what it does?  👀


Can we improve the quality of open LLMs for more languages?

Step 1: Evaluate current SOTA.

The Data Is Better Together community has rated more than 10K prompts for quality. We now want to translate a subset of these to help address the language gap in model evals.

The plan is roughly this:

- We started with https://huggingface.co/datasets/DIBT/10k_prompts_ranked and took a subset of 500 high-quality prompts
- We're asking the community to translate these prompts into different languages
- We'll evaluate the extent to which we can use AlpacaEval and similar approaches to rate the outputs of models across these different languages
- If it works well, we can more easily evaluate open LLMs across different languages by using a judge LLM to rate the quality of outputs from different models.

You can find more details in our new GitHub repo: https://github.com/huggingface/data-is-better-together (don't forget to give it a ⭐!)

dvilasuero clefourrier samusenps sarahooker Joseph717171


I love vision language models 💗
My favorite is KOSMOS-2, because it's a grounded model (it doesn't hallucinate).
In this demo you can,
- ask a question about the image,
- do detailed/brief captioning,
- localize the objects! 🤯
It's just amazing for VLM to return bounding boxes 🤩
Try it here https://huggingface.co/spaces/merve/kosmos2

meryem1232 samusenps osanseviero plmsmile victor GO4code theainerd ShivjiAgnihotri


AV2Wav: Diffusion-Based Re-synthesis from Continuous Self-supervised
  Features for Audio-Visual Speech Enhancement (2309.08030)

🚀🗣🌟 New Research Alert - ICASSP 2024! 🌟🗣🚀
📄 Title: AV2Wav: Diffusion-Based Re-synthesis from Continuous Self-supervised Features for Audio-Visual Speech Enhancement 🌟🚀

📝 Description: Diffused Resynthesis and HuBERT Speech Quality Enhancement.

👥 Authors: Ju-Chieh Chou, Chung-Ming Chien, Karen Livescu

📅 Conference: ICASSP, 14-19 April 2024 | Seoul, Korea 🇰🇷

🔗 Paper: https://huggingface.co/papers/2309.08030

🌐 Web Page: https://home.ttic.edu/~jcchou/demo/avse/avse_demo.html

📚 More Papers: more cutting-edge research presented at other conferences in the
DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin

🚀 Added to the Speech Enhancement Collection: https://huggingface.co/collections/DmitryRyumin/speech-enhancement-65de31e1b6d9a040c151702e

🔍 Keywords: #AV2Wav #SpeechEnhancement #SpeechProcessing #AudioVisual #Diffusion #ICASSP2024 #Innovation

DmitryRyumin samusenps taufiqdp enzostvs osanseviero merve victor Shir02021


Yi: Open Foundation Models by 01.AI (2403.04652)

DeepSeek-VL: Towards Real-World Vision-Language Understanding (2403.05525)

OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist
  Autonomous Agents for Desktop and Web (2402.17553)

Design2Code: How Far Are We From Automating Front-End Engineering? (2403.03163)

Diaries of Open Source. Part 4!

🌏Cohere and Cohere4AI release Command-R, a 35B model that is multilingual, RAG-optimized, and can manage tools!
Model: https://huggingface.co/CohereForAI/c4ai-command-r-v01
Blog post: https://txt.cohere.com/command-r/

🧑🍳StarChat2: A powerful code model that is conversational
Try it out: https://huggingface.co/spaces/HuggingFaceH4/starchat2-playground
Repos: https://huggingface.co/collections/HuggingFaceH4/starchat2-15b-65f068417b330fafad751fce
Training code: https://github.com/huggingface/alignment-handbook/tree/main/recipes/starchat2-15b

🐲Yi-9B: trained on 3 trillion tokens, this english-chinese LLM is quite good and with a very nice detailed report!
Model: https://huggingface.co/01-ai/Yi-9B
Paper: https://huggingface.co/papers/2403.04652

🐋DeepSeek-VL, 1.3B and 7B VLMs
Paper: https://hf.co/papers/2403.05525
Large model: https://huggingface.co/deepseek-ai/deepseek-vl-7b-chat

✍Writer releases OmniACT: a dataset for multimodal agents for desktop and web.
Dataset: https://huggingface.co/datasets/Writer/omniact
Paper: https://huggingface.co/papers/2402.17553

🍎Apple releases MobileCLIP: fast image-text models! https://github.com/apple/ml-mobileclip

🦙💪LlamaGym - fine-tune LLM agents with RL in just a few lines of code! https://github.com/KhoomeiK/LlamaGym

 🖼New multimodal leaderboard ConTextual https://huggingface.co/blog/leaderboard-contextual

🎁 Design2Code: benchmark for multimodal LLMs for automating front-end development.
Dataset https://huggingface.co/datasets/SALT-NLP/Design2Code
Paper https://huggingface.co/papers/2403.03163
Project https://salt-nlp.github.io/Design2Code/

You can find the previous part at https://huggingface.co/posts/osanseviero/633758457910104

samusenps victor kramp DmitryRyumin adamelliotfields merve mohammedbriman giux78 MichaelFried theainerd avinash02 VanshRana12


You gotta love what Aapple’s mlx team cooked:

- A unified memory model that literally does compute-magic: parallel operations with automatic dependency insertions.
- Supports off-the-shelf use of all the fun stuff in composable func transformations (differentiation,  vectorization, computation graph optimization).
- Houses simplified forms of all the APIs we love and in the languages we adore (python, C++, C) sorry Swift :)
- mlx.nn is a stallion 🔥 simple to use.
- Open-source friendly (who would have thought lol).
- Dynamic graph construction👍🏼
- Supports both CPU and GPU🤖
- Beginner Friendly 👌🏼
- Great examples (clean code💯)
- Good documentation

Well done Awni Hannun et al 👏🏼

Could this be The Transformer of ml frameworks? Well at least for us mac users 😂

Repo: https://github.com/ml-explore/mlx
Examples: https://github.com/ml-explore/mlx-examples
Documentation: https://ml-explore.github.io/mlx/build/html/python/nn.html


🚀 Excited to unveil the Augmented ARC-Challenge Dataset with Chain-of-Thought Reasoning! 🧠✨

📚 Created by enhancing the ARC dataset with AI-generated reasoning from Google's Gemini Pro, this resource aims to improve question answering models' ability to tackle complex science queries.

🔍 Features:
- 1068 training examples
- Detailed reasoning steps for nuanced understanding
- Questions spanning physics, chemistry, biology, & more!

🌟 Ideal for benchmarking QA models, enhancing model interpretability, and studying in-context examples.

🔗 Dive in and help your models learn the art of reasoning!

🔎 Explore more: https://huggingface.co/datasets/Locutusque/arc-cot


Stealing Part of a Production Language Model (2403.06634)

Stealing Part of a Production Language Model

https://huggingface.co/papers/2403.06634

We introduce the first model-stealing attack that extracts precise, nontrivial information from black-box production language models like OpenAI's ChatGPT or Google's PaLM-2. Specifically, our attack recovers the embedding projection layer (up to symmetries) of a transformer model, given typical API access. For under \20 USD, our attack extracts the entire projection matrix of OpenAI's Ada and Babbage language models. We thereby confirm, for the first time, that these black-box models have a hidden dimension of 1024 and 2048, respectively. We also recover the exact hidden dimension size of the gpt-3.5-turbo model, and estimate it would cost under 2,000 in queries to recover the entire projection matrix. We conclude with potential defenses and mitigations, and discuss the implications of possible future work that could extend our attack.

ajibawa-2023 taufiqdp arnavgrg adamelliotfields evdcush abdullahalzubaer bootsbootsboots philipp-zettl


Hello World! This post is written by the Large Action Model framework LaVague! Find out more on https://github.com/mithril-security/LaVague

Edit: Here is the video of 🌊LaVague posting this. This is quite meta

samusenps osanseviero abidlabs akhaliq PeepDaSlan9 julien-c eduagarcia diegotluz OmRajani

samusenps abidlabs akhaliq julien-c diegotluz


How Far Are We from Intelligent Visual Deductive Reasoning? (2403.04732)

Can you beat an AI at Raven puzzles?

https://huggingface.co/spaces/HuggingFaceM4/ai_raven

The most powerful vision+language AI systems like Gemini or GPT4V struggle with this problem when used out-of-the-box (https://huggingface.co/papers/2403.04732).

But when properly trained, a small ~8B model can be very accurate at these IQ tests, solely based on visual inputs!

Raven's Progressive Matrices are visual intelligence tests invented in the 1930s designed to measure abstract reasoning and problem-solving ability. The test consists of a series of matrices or patterns with one part missing. The task for the test-taker is to identify the missing piece from a set of options.

Such puzzles can be procedurally generated at scale. https://huggingface.co/datasets/HuggingFaceM4/RAVEN is one example. The complexity of the puzzles is then controlled by the complexity of the generation procedure.

We fine-tuned an early checkpoint of our upcoming vision-and-language model idefics2 on that dataset. The resulting checkpoint yields ~91% accuracy! No chain of thoughts, no pre-processing of the image, no additional inputs or metadata, just the RAVEN problem fed to the model as a standalone image (and a short instruction to the model “Which figure should complete the logical sequence?”), with the training objective being the standard cross-entropy.

Just another evidence that in a lot of cases, for a given well-scoped problem, you will be better off paying to collect & annotate data, and fine-tune a model on that data (i.e. build your own AI) than wastefully trying to solve that problem with a gigantic general-purpose model you call through a paid API!


Can we align code generation models to be good at chat without compromising their base capabilities 🤔?

This was the question the H4 team asked itself when BigCode released StarCoder2 a bit over a week ago. We knew that code models like https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-instruct and https://huggingface.co/m-a-p/OpenCodeInterpreter-DS-33B get impressive scores on code benchmarks like HumanEval, but they tend to score poorly on chat benchmarks like MT Bench and IFEval. We also knew that the Zephyr recipe we applied to Mistral 7B produced a strong chat model, so we wondered -- could be tweaked to produce a strong coding assistant?

It turns out the answer is yes and I'm happy to share StarChat2, a DPO fine-tune of StarCoder2 15B that scores highly on both HumanEval and MT Bench / IFEval 🌟!

The most interesting lesson for me was that you get better models by blending in more code/math data than chat during the SFT step - in terms of tokens, we found a ratio of 3:1 worked best.

Anyway, here's a demo of the model, along with all the code and datasets we used to train it:

* Demo: https://huggingface.co/spaces/HuggingFaceH4/starchat2-playground
* Collection: https://huggingface.co/collections/HuggingFaceH4/starchat2-15b-65f068417b330fafad751fce
* Recipe: https://github.com/huggingface/alignment-handbook

Hope it's useful to others!

lvwerra kashif osanseviero smangrul jordangong Chunte sasikiran MexIvanov vicgalle medmac01


🎥 🤾 Vid2Persona: talk to person from video clip

A fun project over the last week with @sayakpaul. It has a simple pipeline from extracting traits of video characters to chatting with them.

Under the hood, this project leverages the power of both commercial and open source models. We used Google's Gemini 1.0 Pro Vision model to understand the video content directly, then we used HuggingFaceH4/zephyr-7b-beta model to make conversation!

Try it Hugging Face Space and let us know what you think.
: https://huggingface.co/spaces/chansung/vid2persona

The space application is a dedicated implementation for ZeroGPU environment + Hugging Face Inference API with PRO account. If you wish to host it on your own environment, consider duplicate the space or run locally with the project repository
: https://github.com/deep-diver/Vid2Persona

chansung anonyme789 Chan-Y victor osanseviero xprilion

anonyme789 Srulikbd osanseviero xprilion


If you're trying to run MoE Mixtral-8x7b under DeepSpeed w/ HF Transformers it's likely to hang on the first forward.

The solution is here https://github.com/microsoft/DeepSpeed/pull/4966?_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=en-US#issuecomment-1989671378

and you need deepspeed>=0.13.0

Thanks to Masahiro Tanaka for the fix.

nss-ysasaki sitloboi2012 theainerd osanseviero victor muhtasham hangzhang-nlp


Relightable and Animatable Neural Avatars from Videos (2312.12877)

🚀🕺🌟 New Research Alert - AAAI 2024! 🌟💃🚀
📄 Title: Relightable and Animatable Neural Avatars from Videos 🌟🚀

📝 Description: Relightable & animatable neural avatars from sparse videos.

👥 Authors: Wenbin Lin, Chengwei Zheng, Jun-Hai Yong, and Feng Xu

📅 Conference: AAAI, February 20-27, 2024 | Vancouver, Canada 🇨🇦

🔗 Paper: https://huggingface.co/papers/2312.12877

🌐 Github Page: https://wenbin-lin.github.io/RelightableAvatar-page
📁 Repository: https://github.com/wenbin-lin/RelightableAvatar

📺 Video: https://www.youtube.com/watch?v=v9rlys0xQGo

📚 More Papers: more cutting-edge research presented at other conferences in the https://huggingface.co/spaces/DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin

🚀 Added to the Avatars Collection: https://huggingface.co/collections/DmitryRyumin/avatars-65df37cdf81fec13d4dbac36

📚 Added to the AAAI 2024 Papers: https://github.com/DmitryRyumin/AAAI-2024-Papers

🔍 Keywords: #NeuralAvatar #RelightableAvatars #AnimatableAvatars #3DModeling #PhotorealisticRendering #ShadowModeling #DigitalAvatars #GeometryModeling #AAAI2024 #DeepLearning #Animation #Innovation

DmitryRyumin samusenps osanseviero sitloboi2012 Lician


Gemini 1.5: Unlocking multimodal understanding across millions of tokens
  of context (2403.05530)

671 authors 🤯

https://huggingface.co/papers/2403.05530 Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

osanseviero muhtasham dvilasuero pabloce diwank


New foundation model on document understanding and generation in  transformers 🤩
UDOP by MSFT is a bleeding-edge model that is capable of many tasks, including question answering, document editing and more! 🤯
Demo 👉 https://huggingface.co/spaces/merve/UDOP
It is a model that combines vision, text and layout. 📝
This model is very interesting because the input representation truly captures the nature of the document modality: text, where the text is, and the layout of the document matters!
If you know T5, it resembles that: it's pre-trained on both self-supervised and supervised objectives over text, image and layout.
To switch between tasks, one simply needs to change the task specific prompt at the beginning, e.g. for QA, one prepends with Question answering.
As for the architecture, it's like T5, except it has a single encoder that takes in text, image and layout, and two decoders (text-layout and vision decoders) combined into one.
The vision decoder is a masked autoencoder (thus the capabilities of document editing).
For me, the most interesting capability is document reconstruction, document editing and layout re-arrangement. This decoder isn't released though because it could be used maliciously to fake document editing.
Overall, the model performs very well on document understanding benchmark (DUE) and also information extraction (FUNSD, CORD) and classification (RVL-CDIP) for vision, text, layout modalities.
You can learn more about the model from below resources (h/t to
@nielsr), thanks a lot for reading 🤗
Docs: https://huggingface.co/docs/transformers/main/en/model_doc/udop 📚
Checkpoints: https://huggingface.co/collections/microsoft/udop-65e625124aee97415b88b513
Demo notebooks: https://github.com/NielsRogge/Transformers-Tutorials/tree/master/UDOP 📕

samusenps rajistics not-lain cstr diwank


Information Flow Routes: Automatically Interpreting Language Models at
  Scale (2403.00824)

🔍 Today's pick in Interpretability & Analysis of LMs: Information Flow Routes: Automatically Interpreting Language Models at Scale by @javifer @lena-voita

This work presents a novel method to identify salient components in Transformer-based language models by decomposing the contribution of various model components into the residual stream.

This method is more efficient and scalable than previous techniques such as activation patching, as it only requires a single forward pass through the model to identify critical information flow paths. Moreover, it can be applied without a contrastive template, which is observed to produce results dependent on the selected contrastive example for activation patching.

Information flow routes are applied to Llama 2, showing that:

1. Models show “typical” information flow routes for non-content words, while content words don’t exhibit such patterns.
2. Feedforward networks are more active in the bottom layers of the network (where e.g. subject enrichment is performed) and in very last layer.
3. Positional and subword-merging attention heads are among the most active and important throughout the network.
4. Periods can be treated by the model as BOS tokens by leaving their residual representation mostly untouched during the forward pass.

Finally, the paper also demonstrates that some model components are specialized for specific domains, such as coding or multilingual texts, suggesting a high degree of modularity in the network. The contribution of domain-specific heads obtained by projecting right singular values of the OV circuit to the unembedding matrix show highly interpretable concepts being handled in granular model components.

📄 Paper: https://huggingface.co/papers/2403.00824

🔍 All daily picks: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9

Kukedlc samusenps javifer CristianJD Theli


ShortGPT: Layers in Large Language Models are More Redundant Than You
  Expect (2403.03853)

A recent paper titled "ShortGPT: Layers in Large Language Models are More Redundant Than You Expect" proposes a simple and effective approach to pruning Large Language Models (LLMs) by removing redundant layers.

Key points:
* Discovers significant redundancy across layers in LLMs, with some layers playing a negligible role for the final performance.
* Defines a new metric called Block Influence (BI) to quantify the importance of each layer in an LLM.
* Removes layers with low BI scores, achieving up to 25% reduction in parameters and computation while maintaining 92% of the LLM's performance.

Congrats to the authors for their work!

Paper: https://huggingface.co/papers/2403.03853


ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment (2403.05135)

ELLA

Equip Diffusion Models with LLM for Enhanced Semantic Alignment

https://huggingface.co/papers/2403.05135

Diffusion models have demonstrated remarkable performance in the domain of text-to-image generation. However, most widely used models still employ CLIP as their text encoder, which constrains their ability to comprehend dense prompts, encompassing multiple objects, detailed attributes, complex relationships, long-text alignment, etc. In this paper, we introduce an Efficient Large Language Model Adapter, termed ELLA, which equips text-to-image diffusion models with powerful Large Language Models (LLM) to enhance text alignment without training of either U-Net or LLM. To seamlessly bridge two pre-trained models, we investigate a range of semantic alignment connector designs and propose a novel module, the Timestep-Aware Semantic Connector (TSC), which dynamically extracts timestep-dependent conditions from LLM. Our approach adapts semantic features at different stages of the denoising process, assisting diffusion models in interpreting lengthy and intricate prompts over sampling timesteps. Additionally, ELLA can be readily incorporated with community models and tools to improve their prompt-following capabilities. To assess text-to-image models in dense prompt following, we introduce Dense Prompt Graph Benchmark (DPG-Bench), a challenging benchmark consisting of 1K dense prompts. Extensive experiments demonstrate the superiority of ELLA in dense prompt following compared to state-of-the-art methods, particularly in multiple object compositions involving diverse attributes and relationships.

vladbogo Dlbk samusenps ClayFace Zmu melohux Benson


𝐑𝐮𝐧𝐬 𝐟𝐚𝐬𝐭 𝐚𝐥𝐬𝐨 𝐨𝐧 𝐂𝐏𝐔!

𝐑𝐮𝐧𝐬 𝐟𝐚𝐬𝐭 𝐚𝐥𝐬𝐨 𝐨𝐧 𝐂𝐏𝐔!

Super work from @DeepMount00:

🚀 𝐃𝐢𝐬𝐜𝐨𝐯𝐞𝐫 𝐔𝐧𝐢𝐯𝐞𝐫𝐬𝐚𝐥 𝐍𝐞𝐫: 𝐀 𝐆𝐥𝐢𝐍𝐞𝐫-𝐁𝐚𝐬𝐞𝐝 𝐈𝐭𝐚𝐥𝐢𝐚𝐧 𝐍𝐄𝐑

Introducing 𝐔𝐧𝐢𝐯𝐞𝐫𝐬𝐚𝐥 𝐍𝐞𝐫 𝐟𝐨𝐫 𝐈𝐭𝐚𝐥𝐢𝐚𝐧 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞, a revolutionary Named Entity Recognition (NER) model evolved from the GliNer architecture and meticulously tailored for the Italian language. This advanced model is a beacon of efficiency and versatility, engineered to 𝐫𝐞𝐜𝐨𝐠𝐧𝐢𝐳𝐞 𝐚𝐧𝐲 𝐞𝐧𝐭𝐢𝐭𝐲 𝐭𝐲𝐩𝐞 within the rich nuances of Italian, using a bidirectional transformer encoder. It stands out as an ideal solution for those navigating the challenges of resource-limited environments or seeking an efficient alternative to the cumbersome Large Language Models (LLMs).
𝐑𝐮𝐧𝐬 𝐟𝐚𝐬𝐭 𝐚𝐥𝐬𝐨 𝐨𝐧 𝐂𝐏𝐔!

Experience this Italian-focused innovation live on Hugging Face Spaces:
https://huggingface.co/spaces/DeepMount00/universal_ner_ita

Paper:  https://arxiv.org/abs/2311.08526 Urchade Zaratiana et all. great work!

samusenps osanseviero Dlbk DeepMount00 urchade giux78 tomaarsen MoritzLaurer


Good time to drop some 🤗 in the comments  > https://x.com/elonmusk/status/1767108624038449405

samusenps Dlbk sa8 frntn clefourrier Zmu CKeibel lvalue clem


Some papers deserve a standing ovation after reading, “Direct Preference Optimization: Your Language Model is Secretly a Reward Model” is one such paper:

One major drawback of LLMs is the lack of precise control over their behavior which makes it very difficult to align with desired outcomes. Existing methods to mitigate this involves gathering generated, humanly labeled data and fine-tuning the unsupervised LLM to align with preferences - this is known as Reinforcement Learning From Human Feedback (RLHF).

RLHF is an incredibly complex, usually unstable and computationally costly method. It involves first scaling a suitable reward model that meets human preferences then fine-tuning the language model with RL to maximize the estimated reward while maintaining a major part of the original model.

This paper introduces a new algorithm called Direct Preference Optimization (DPO) that simplifies the whole process. In short, it directly optimizes the LM without explicit reward modeling or reinforcement learning.  This is achieved by leveraging a mapping between reward functions and optimal policies, allowing the constrained reward maximization problem to be optimized exactly with a single stage of policy training.

DPO’s genius lies in its ability to intuitively increase the relative log probability of preferred to "unpreferred" responses.

The amazing thing about this paper is how fundamentally self-proven it is - from clearly stating the problem to explicitly explaining the underlying theory backed with mathematical proofs, it’s just genius.

In my opinion, every academic research paper should follow this approach.  It won the 2023 NeurIPS Outstanding paper award (Category: Outstanding Main Track Runner-Ups).

osanseviero victor stonestack kawinm Theli AI-B


🚀⭐Introducing our new survey "Leveraging Biomolecule and Natural Language through Multi-Modal Learning: A Survey"

arxiv: https://arxiv.org/abs/2403.01528
github: https://github.com/QizhiPei/Awesome-Biomolecule-Language-Cross-Modeling

The integration of biomolecular modeling with natural language (BL) has emerged as a promising interdisciplinary area at the intersection of artificial intelligence, chemistry and biology. This approach leverages the rich, multifaceted descriptions of biomolecules contained within textual data sources to enhance our fundamental understanding and enable downstream computational tasks such as biomolecule property prediction. The fusion of the nuanced narratives expressed through natural language with the structural and functional specifics of biomolecules described via various molecular modeling techniques opens new avenues for comprehensively representing and analyzing biomolecules. By incorporating the contextual language data that surrounds biomolecules into their modeling, BL aims to capture a holistic view encompassing both the symbolic qualities conveyed through language as well as quantitative structural characteristics. In this review, we provide an extensive analysis of recent advancements achieved through cross modeling of biomolecules and natural language.

QizhiPei osanseviero victor samusenps MaziyarPanahi bczhou


CAMixerSR: Only Details Need More "Attention" (2402.19289)

🚀🖼🌟 New Research Alert - CVPR 2024! 🌟🖼🚀
📄 Title: CAMixerSR: Only Details Need More "Attention" 🌟🚀

📝 Description: CAMixerSR is a new approach integrating content-aware accelerating framework and token mixer design, to pursue more efficient SR inference via assigning convolution for simple regions but window-attention for complex textures. It exhibits excellent generality and attains competitive results among state-of-the-art models with better complexity-performance trade-offs on large-image SR, lightweight SR, and omnidirectional-image SR.

👥 Authors: Yan Wang, Shijie Zhao, Yi Liu, Junlin Li, and Li Zhang

📅 Conference: CVPR, Jun 17-21, 2024 | Seattle WA, USA 🇺🇸

🔗 Paper: https://huggingface.co/papers/2402.19289

🔗 Repository: https://github.com/icandle/CAMixerSR

📚 More Papers: more cutting-edge research presented at other conferences in the https://huggingface.co/spaces/DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin

🚀 Added to the Image Enhancement Collection: https://huggingface.co/collections/DmitryRyumin/image-enhancement-65ee1cd2fe1c0c877ae55d28

🔍 Keywords: #CAMixerSR #SuperResolution #WindowAttention #ImageEnhancement #CVPR2024 #DeepLearning #Innovation

DmitryRyumin samusenps osanseviero Jakaline victor Dimo87


Quantize 7B paramater models in 60 seconds using Half Quadratic Quantization (HQQ).

This game-changing technique allows for rapid quantization of models like Llama-2-70B in under 5 minutes, outperforming traditional methods by 50x in speed and offering high-quality compression without calibration data.

Mobius Labs innovative approach not only significantly reduces memory requirements but also enables the use of large models on consumer-grade GPUs, paving the way for more accessible and efficient machine learning research.

Mobius Labs' method utilizes a robust optimization formulation to determine the optimal quantization parameters, specifically targeting the minimization of errors between original and dequantized weights. This involves employing a loss function that promotes sparsity and utilizes a non-convex lp<1-norm, making the problem challenging yet solvable through a Half-Quadratic solver.

This solver simplifies the problem by introducing an extra variable and dividing the optimization into manageable sub-problems. Their implementation cleverly fixes the scale parameter to simplify calculations and focuses on optimizing the zero-point, utilizing closed-form solutions for each sub-problem to bypass the need for gradient calculations.

Check out the colab demo where you are able to quantize models (text generation and multimodal) for use with vLLM or Timm backend as well as transformers!

AutoHQQ: 👉 https://colab.research.google.com/drive/1cG_5R_u9q53Uond7F0JEdliwvoeeaXVN?usp=sharing
Code: https://github.com/mobiusml/hqq
HQQ Blog post: https://mobiusml.github.io/hqq_blog/

Edit: Here is an example of how powerful HQQ can be: https://huggingface.co/macadeliccc/Nous-Hermes-2-Mixtral-8x7B-DPO-HQQ

Citations:

@misc{badri2023hqq,
	title = {Half-Quadratic Quantization of Large Machine Learning Models},
	url = {https://mobiusml.github.io/hqq_blog/},
	author = {Hicham Badri and Appu Shaji},
	month = {November},
	year = {2023}
}

Warung appoose boqiangliang hiepxanh Lewdiculous cnmoro MH0386


Over the past week, I've been putting Claude through its paces, focusing primarily on productivity tasks (you know, the good old BAU – Business As Usual).

1. Python/Torch/Transformers/AI/ML
Right off the bat, I threw some complex AI/ML tasks at Claude, and I must say, it handled them with finesse. It even caught a few things that GPT missed! However, let's not get too carried away – we're not quite at the auto-code level just yet.

2. Brainstorming
This is where Claude falls a bit short. It seems to be more grounded than its competitors, which might not be ideal for generating novel ideas. If you're looking for a brainstorming partner, you might want to look elsewhere.

3. Attention
Despite the claims of super-large attention in the paper, Claude's "forgetting" mechanism seems to be more pronounced. It tends to miss entire chunks of information rather than just specific details like GPT does.

4. Following / Tasks
I hit a roadblock when Claude couldn't generate a LaTeX document. It's not the best at following complex, multi-step tasks.

5. Hallucinations
Oh boy, does Claude hallucinate! And when it does, it's on a whole new level of nonsense. The hallucinations seem to align with its grounded nature, making them even more convincing within the context of the prompt.

6. Sycophancy
Claude is quite the people-pleaser. I've found that using an adversarial brainstorming approach is more beneficial and time-efficient, as it forces me to highlight Claude's mistakes rather than letting it focus on being a sweet, pleasant minion.

7. Interface / UI
There's definitely room for improvement here. Basic features like stepping back on a prompt and stopping generation with the ESC key are missing. These are essential for extracting and composing content effectively.

Despite these limitations, I firmly believe that Claude is currently the #1

samusenps macadeliccc osanseviero victor delano Nymbo

Иванов Константин


LAST CHANCE TO TRY 🌟STARCODER2

After today it's gone !

actually not - just joking ! it's <3 open source !

 just trying to get folks' attention to my featured "Spaces of the Week" :

https://huggingface.co/spaces/Tonic/starcoder2

drop a like for your boy and join us next week for making fine tunes !


Multi-Instance Generation Controller: Enjoy complete control over position generation, attribute determination, and count!

code link: https://github.com/limuloo/MIGC
project page: https://migcproject.github.io/

MIGC decouples multi-instance generation into individual single-instance generation subtasks within the cross-attention layer of Stable Diffusion.

Welcome to follow our project and use the code to create anything you imagine!

Please let us know if you have any suggestions!

Flowerfan limuloo1999 little-air-dust LeyRio xiaodi sanaka87 wenhaozzu yqqCheergo Whalesong Csplk mingzs2 samusenps taufiqdp Jouryjc osanseviero Warung George-Blaze danniel2023 linoyts Jakaline ariarcus erickdp radames

limuloo1999 xiaodi sanaka87 Shinku Whalesong mingzs2 samusenps vvvvJ osanseviero danniel2023 linoyts shashankbodapati radames jeffboudier


Updating PaperQA Gradio app and Hugging Face Space.
: Link ➡ https://huggingface.co/spaces/chansung/paper_qa
: Standalone repo ➡ https://github.com/deep-diver/paperqa-ui

The final goal is to let ppl have their own paper archive. At the end, You will be able to easily *clone* on local or Hugging Face Space with Google's Gemini API Key (which is free), Hugging Face Access Token. You can just drop arXiv IDs at the bottom, then all the auto analyze papers are automatically archived on Hugging Face Dataset repo.

Here are few updates included, and dig in the source code if you want similar features for your use cases!
🖥 making complex UI + fully responsive
+ making UI as quickly as possible (avoid server-client when possible)
💬 Permanent Chat history management with in-browser local storage
+ Chat history management *per* paper
+ Chat history management in lazy mode (too many paper, impossible to create chat history for every single paper beforehand, hence)

Current plan is to support Gemini and any open source models on Hugging Face PRO account, but will expand it to GPT4 soon.

Any suggestion on this project is welcome! possibly,
- hooking up RAG system (open models' context length is small)
- hooking up Internet search system
- image/figure analysis
....


Keep stacking cool stuff and getting better results! After I changed the standard vision encoder to SigLIP, NLLB-CLIP got a 10% average performance improvement. And now, I added matryoshka layers (https://arxiv.org/abs/2205.13147) to enable smaller embeddings and got another 6% performance boost! Plus, thanks to MRL, 4.5x smaller embeddings retain 90%+ quality.

The large model is finally SoTA for both image and text multilingual retrieval!

The models are available on the hub:
- https://huggingface.co/visheratin/nllb-siglip-mrl-base
- https://huggingface.co/visheratin/nllb-siglip-mrl-large


Retrieval-Augmented Generation (RAG)
Redeemer of the "hallucination problem"

It is fair enough to argue that "hallucinations" in LLMs are just mere reflections of what we humans occasionally do - well it gets worse as we get older, but these models are brain inspired, thus such behaviors are likely inherently unavoidable. After all, we are just dreamers trying make sense of this life.

The best we can do is minimize and control it - but humanly how? By first feeding on relevant facts and then developing a habit that allows us to easily access those facts when needed. This is what RAG is all about - it's just a control mechanism that keeps the LLM aligned with reality and fact.

But How Does RAG Work?

Well, to some extent it is domain-specific but the overall workflow boils down to the following:

1. It makes use of a retrieval mechanism that hunts for facts relevant to a query - this involves an end-to-end backpropagation that leverages a retriever (Query Encoder + Document Index or Source of Truth) with a pre-trained generative model.

2. The generative model then uses the facts retrieved, performs some verification to give a more accurate response.

To summarize, the RAG architecture houses a pre-existing knowledge source model (termed parametric memory), which then utilizes a Source-of-Truth model or vector indexed data (termed non-parametric memory) that is accessed by a pre-trained neural retriever, in order to produce more informed, contextually appropriate and factually correct responses.

Sort of a "Genius Engine" if you might say. If only we humans could harness such, AGI would be much much sooner lol.

In the meantime, I have been Jaward Sesay (Chinese name 苏杰 Sujie) - a young Sierra Leonean, aspiring AI Researcher. I like to read, share and try implementing AI research papers. Also like dunking on big tech while rooting for open-source. My mentor @karpathy, I dream of him following me back on X lol. Thanks.

samusenps osanseviero paralym Jafta rohitdavas


GRID-6X : Layout for Seamless Image Assembly 🔥

🪨Demo:  https://huggingface.co/spaces/prithivMLmods/GRID-6X
🪨Doc / Blog: https://huggingface.co/blog/prithivMLmods/grid-6x

In the `infer` function:
```python
grid_img = Image.new('RGB', (width * grid_size_x, height * grid_size_y))
for i, img in enumerate(result.images[:num_images]):
    grid_img.paste(img, (i % grid_size_x * width, i // grid_size_x * height))
```
1. **Image Initialization**: `grid_img` is a blank canvas that will hold the images in a grid format.
2. **Image Placement**: Images are pasted onto the canvas using a loop:
   - **Horizontal Position**: `(i % grid_size_x) * width` calculates the x-coordinate.
   - **Vertical Position**: `(i // grid_size_x) * height` calculates the y-coordinate.

1. **Grid Size Selection**: The user selects the grid size from options like "2x1", "1x2", "2x2", "2x3", "3x2", and "1x1". Each option corresponds to the arrangement of images:
   - **2x1**: 2 images in a single row
   - **1x2**: 1 image in two rows (column layout)
   - **2x2**: 2 rows with 2 images each
   - **2x3**: 2 rows with 3 images each
   - **3x2**: 3 rows with 2 images each
   - **1x1**: A single image (default)

2. **Image Generation**: Based on the grid size selection, the app calculates the number of images to generate. For example:
   - If the grid size is "2x2", the app generates 4 images.
   - For "3x2", it generates 6 images.

-> Each option arranges images accordingly, providing flexibility in viewing multiple images in one output.

-> Added both of these spaces that support the GRID functionality Layout for Seamless Image Assembly :

----------
🔥IMAGINEO-4K: https://huggingface.co/spaces/prithivMLmods/IMAGINEO-4K

🔥GRID-6X: https://huggingface.co/spaces/prithivMLmods/GRID-6X
----------
.
.
.@prithivMLmods  🤗

rdrede darksfx hypergod civet789 prithivMLmods


#EMNLP2024 is happening soon! Unfortunately, I will not be on site, but I will present our poster virtually on Wednesday, Nov 13 (7:45 EST / 13:45 CEST) in Virtual Poster Session 2.

In this work, we leverage self-training in an active learning loop in order to train small language models with even less data. Hope to see you there!


Hello everyone!!! I am new to this and a little out of my depth (aLOT out of my depth!! LOL!) I am looking through the site and wanted to ask if there were any quailty primers i should read? or a good basic getting started post?

Thanks in advance!!


Style flo : : 🎉🤗

{ Try Now on Flux LoRA DLC ⛵ } : https://huggingface.co/spaces/prithivMLmods/FLUX-LoRA-DLC

-- Undersea
{ Red Fluid } : https://huggingface.co/prithivMLmods/Red-Undersea-Flux-LoRA

-- 3D Realmix
{ 3D Portrait Render } : https://huggingface.co/prithivMLmods/3D-Render-Flux-LoRA

-- Pop
{ Yellow Pop } : https://huggingface.co/prithivMLmods/Yellow-Pop-Flux-Dev-LoRA

-- Grid
{ Purple Grid } : https://huggingface.co/prithivMLmods/Purple-Grid-Flux-LoRA

{ collections : : }

🚀 Flux LoRA :
https://huggingface.co/collections/prithivMLmods/flux-lora-collections-66dd5908be2206cfaa8519be

🚀Collection zero: https://huggingface.co/collections/prithivMLmods/collection-zero-and-demo-recently-updated-65e48a7dd8212873836ceca2

.
.
@prithivMLmods  🧨


Pleased to announce Cat1.0, the newest iteration of my roleplay fine-tunes! https://huggingface.co/rwitz/cat1.0
The model is fine-tuned from Llama-3.1 8B on VERY high quality roleplay chat logs, each stretching for thousands of tokens. Also excels at logic, especially in conversational reasoning! Feel free to give it a test!


How To Use Mochi 1 Open Source Video Generation Model On Your Windows PC, RunPod and Massed Compute

Tutorial Link : https://youtu.be/iqBV7bCbDJY

Mochi 1 from Genmo is the newest state-of-the-art Open Source video generation model that you can use for free on your computer. This model is a breakthrough like the very first Stable Diffusion model but this time it is starting for the video generation models. In this tutorial, I am going to show you how to use Genmo Mochi 1 video generation model on your computer, on windows, locally with the most advanced and very easy to use SwarmUI. SwarmUI as fast as ComfyUI but also as easy as using Automatic1111 Stable Diffusion web UI. Moreover, if you don’t have a powerful GPU to run this model locally, I am going to show you how to use this model on the best cloud providers RunPod and Massed Compute.

🔗 Public Open Access Article Used in Video ⤵
▶ https://www.patreon.com/posts/106135985

Amazing Ultra Important Tutorials with Chapters and Manually Written Subtitles / Captions
Stable Diffusion 3.5 Large How To Use Tutorial With Best Configuration and Comparison With FLUX DEV : https://youtu.be/-zOKhoO9a5s

FLUX Full Fine-Tuning / DreamBooth Tutorial That Shows A Lot Info Regarding SwarmUI Latest : https://youtu.be/FvpWy1x5etM

Full FLUX Tutorial — FLUX Beats Midjourney for Real : https://youtu.be/bupRePUOA18

Main Windows SwarmUI Tutorial (Watch To Learn How to Use)

How to install and use. You have to watch this to learn how to use SwarmUI
Has 70 chapters and manually fixed captions : https://youtu.be/HKX8_F1Er_w

MonsterMMORPG DIEUDONNE972 Svngoku aleph65 andmev Yingchaooo mikeehren


📢 Have you ever been wondered how specifically Transformers were capable for handling long input contexts?
I got a chance to tackle this through long document texts summarization problem, and delighted to share the related survey and diagram for a quick skimming below:

Preprint 📝 https://nicolay-r.github.io/website/data/preprint-AINL_2023_longt5_summarization.pdf
Springer 📝 https://link.springer.com/article/10.1007/s10958-024-07435-z

🎯 The aim of the survey was the development of the long-document summarizer for mass-media news in Vietnamese language. 🇻🇳

Sharing for a quick skimming of the methods performance overview of various LM-based solution across several datasets, covering domain-oriented advances in Vietnamese language (see attached screenshots)

As for solution we consider:
☑ 1. Adapt existed https://huggingface.co/google/pegasus-cnn_dailymail for summarizing large dataset for arranging training
☑ 2. Tuning https://huggingface.co/google/long-t5-tglobal-large  suitable for performing generative summarization.

Implementation details:
🌟 https://github.com/nicolay-r/ViLongT5
(Simplier to go with huggingface rather flaxformer that so far become a legacy engine)


Last Week in Medical AI: Top Research Papers/Models 🔥
(November 2 -November 9, 2024)

🏅 Medical AI Paper of the Week:
Exploring Large Language Models for Specialist-level Oncology Care

Medical LLM & Other Models:
- GSCo: Generalist-Specialist AI Collaboration
- PediatricsGPT: Chinese Pediatric Assistant
- MEG: Knowledge-Enhanced Medical QA
- AutoProteinEngine: Multimodal Protein LLM

Frameworks and Methodologies:
- BrainSegFounder: 3D Neuroimage Analysis
- PASSION: Sub-Saharan Dermatology Dataset
- SAM for Lung X-ray Segmentation
- Label Critic: Data-First Approach
- Medprompt Runtime Strategies

Medical LLM Applications:
- CataractBot: Patient Support System
- CheX-GPT: X-ray Report Enhancement
- CardioAI: Cancer Cardiotoxicity Monitor
- HealthQ: Healthcare Conversation Chain
- PRObot: Diabetic Retinopathy Assistant

Medical LLMs & Benchmarks:
- MediQ: Clinical Reasoning Benchmark
- Touchstone: Segmentation Evaluation
- Medical LLM Adaptation Progress
- Fine-Tuning Medical QA Strategies

AI in Healthcare Ethics:
- Healthcare Robotics with LLMs
- XAI in Clinical Practice
- Precision Rehabilitation Framework
- Multimodal AI Challenges

Now you can watch and listen to the latest Medical AI papers daily on our YouTube and Spotify channels as well!

- Full Thread: https://x.com/OpenlifesciAI/status/1855207141302473090
- YouTube: https://youtu.be/ad0uTnYuTo8
- Spotify: https://open.spotify.com/episode/6s39t1UJZk1i10szuXP2qN

aaditya BayesTensor Healthtensor ai-everyday


Thinking about upgrading from Python 3.10 to 3.11? Here's why you should make the move - a deep technical breakdown that might convince you:

>> Performance Revolution
The performance improvements are staggering, with benchmarks showing 10-60% faster execution across different workloads. Let me break down the game-changing features:

>> Core Architecture Changes
Python 3.11's interpreter now uses statically allocated core modules, eliminating the multi-step loading process we've dealt with in 3.10. This means your applications will start 10-15% faster out of the gate.

>> Function Optimization
The redesigned frame objects are a thing of beauty - they've been stripped of unnecessary baggage, resulting in a 3-7% speedup for all function calls. But it gets better: function calls are now inlined, giving us a 1-3% boost, with recursive functions like Fibonacci seeing up to 1.7x improvement!

>> Adaptive Intelligence
The new Specializing Interpreter is perhaps the most exciting addition. Think of it as a lightweight JIT - it identifies hot code paths and optimizes them automatically.

The interpreter now automatically specializes math operations, array indexing, and even sequence unpacking based on actual usage patterns.

>> Exception Handling Revolution
My favorite feature? Zero-cost exceptions! Your try-except blocks no longer carry overhead when no exceptions occur. The code runs at full speed until an exception actually happens.

Ready to make the switch? These improvements aren't just numbers - they're real-world performance gains waiting to be unlocked in your codebase.


Dear AI developers, have you managed to get any grants or compute powers for your research or startup?
If yes, from where and how?


🚀 Child Safe Chatbot (Experimental)Based on google/shieldgemma-9b. Currently in the experimental phase, this configuration could be a starting point to consolidate an LLM that meets the requirements to be safe for minors.
👉 https://huggingface.co/spaces/nmarafo/Child-Safe-Chatbot
#EdTech #ChildSafety #AI #Education #Experimental


🚀 Introducing ColFlor: An Efficient, OCR-Free Vision-Language Document Retrieval Model 🌟

Earlier this year, ColPali revolutionized document retrieval by eliminating the need for error-prone OCR pipelines. Instead, it directly processes the document images. However, with its 3 billion parameters, ColPali is computationally heavy for large-scale applications.

That’s where ColFlor comes in—a smaller, faster alternative! 🎉 At 17x smaller than ColPali, ColFlor offers a more efficient, OCR-free document retrieval solution, making it ideal for users with limited computing resources (GPU Poor). 💡

Key Highlights:
🧠 174M parameters (vs. 3B for ColPali)
⚡ 9.8x faster query encoding, 5.25x faster image encoding
📉 Only 1.8% performance drop on text-rich English documents

Check out the full blog post for more insights on modeling, training, and evaluations across various document retrieval tasks! 🚀
Also, feel free to try our demo on huggingface 🤗

🔗 Resources:
📄 Blog post: https://huggingface.co/blog/ahmed-masry/colflor
🧠 Model: https://huggingface.co/ahmed-masry/ColFlor
💻 Demo: https://huggingface.co/spaces/ahmed-masry/ColFlor-Demo
🏋♂ Training code: https://github.com/AhmedMasryKU/colflor
📊 Evaluation code: https://github.com/AhmedMasryKU/vidore-benchmark-colflor


I built a Hyper Dimensional Computing (HDC) Encoder and Decoder and I also built out all of the code needed to replace the Encoder and Decoder of a Llama model with this HDC model, then train the Llama model on the HDC Encoder/Decoder. All MIT licensed. Here is a video where I break it all down. I can answer any questions about this project or help anyone out where I can. I am not a super developer or anything and I don't have access to enough compute to train this on a large dataset: https://youtu.be/4VsZpGaPK4g


Here is how we can calculate the size of any LLM model:

Each parameter in LLM models is typically stored as a floating-point number. The size of each parameter in bytes depends on the precision.

32-bit precision: Each parameter takes 4 bytes.
16-bit precision: Each parameter takes 2 bytes

To calculate the total memory usage of the model:
Memory usage (in bytes) = No. of Parameters × Size of Each Parameter

For example:
32-bit Precision (FP32)
In 32-bit floating-point precision, each parameter takes 4 bytes.
Memory usage in bytes = 1 billion parameters × 4 bytes
1,000,000,000 × 4 = 4,000,000,000 bytes
In gigabytes: ≈ 3.73 GB

16-bit Precision (FP16)
In 16-bit floating-point precision, each parameter takes 2 bytes.
Memory usage in bytes = 1 billion parameters × 2 bytes
1,000,000,000 × 2 = 2,000,000,000 bytes
In gigabytes: ≈ 1.86 GB

It depends on whether you use 32-bit or 16-bit precision, a model with 1 billion parameters would use approximately 3.73 GB or 1.86 GB of memory, respectively.

zzlxv John6666 danielus Ninjanimus san122 SporkySporkness AtAndDev adriszmar


🎓 Introducing Ukr-lit.com.ua Presentations Dataset - https://huggingface.co/datasets/nyuuzyou/ukr-lit

Dataset highlights:
- 18,001 presentations from ukr-lit.com.ua, a platform for storing and viewing presentations covering a wide range of subjects in Ukrainian school education
- Primarily in Ukrainian, with some Russian and English content
- Each entry includes: URL, title, download URL, filepath, and extracted text content (where available)
- Contains original PPT/PPTX files in addition to metadata
- Data covers a broad spectrum of educational topics and subjects taught in Ukrainian schools
- Dedicated to the public domain under Creative Commons Zero (CC0) license

The dataset can be used for analyzing educational presentation content across various subjects in Ukrainian and other languages, text classification tasks, and information retrieval systems. It's particularly valuable for examining trends in Ukrainian school education, teaching methodologies, and presentation materials used across different academic disciplines. The inclusion of original files allows for in-depth analysis of presentation formats and structures commonly used in Ukrainian educational settings, providing insights into the diverse range of subjects and teaching approaches in the Ukrainian school system.


🌍 Super cool visualization of global PUT requests to Hugging Face over 24 hours, coded by object size, thanks to @port8080!

We're putting this analysis to work to help us architect a more geo-distributed system for the HF storage backend.

Originally shared on LinkedIn: https://www.linkedin.com/posts/ajitbanerjee_one-of-the-joys-of-working-on-the-xethub-activity-7252688424732614656-tFGD

nyuuzyou port8080 Awhildy AtAndDev John6666 gabrielmbmb not-lain yjernite xi0v jsulz


Big news! You can now build strong ML models without days of human labelling

You simply:
- Define your dataset, including annotation guidelines, labels and fields
- Optionally label some records manually.
- Use an LLM to auto label your data with a human (you? your team?) in the loop!

Get started with this blog post:
https://huggingface.co/blog/sdiazlor/custom-text-classifier-ai-human-feedback


You can now build a custom text classifier without days of human labeling!

👍 LLMs work reasonably well as text classifiers.
👎 They are expensive to run at scale and their performance drops in specialized domains.

👍 Purpose-built classifiers have low latency and can potentially run on CPU.
👎 They require labeled training data.

Combine the best of both worlds: the automatic labeling capabilities of LLMs and the high-quality annotations from human experts to train and deploy a specialized model.

Blog: https://huggingface.co/blog/sdiazlor/custom-text-classifier-ai-human-feedback

dvilasuero AtAndDev NickoSELI tiendung Darklight0012 davidberenstein1957 clem

nyuuzyou John6666 victor Ninjanimus AtAndDev tiendung davidberenstein1957


By far the coolest release of the day!
> The Open LLM Leaderboard, most comprehensive suite for comparing Open LLMs on many benchmarks, just released a comparator tool that lets you dig into the detail of differences between any models.

Here's me checking how the new Llama-3.1-Nemotron-70B that we've heard so much compares to the original Llama-3.1-70B. 🤔🔎

Try it out here 👉 https://huggingface.co/spaces/open-llm-leaderboard/comparator


Finetuned Multimodal Language Models Are High-Quality Image-Text Data
  Filters (2403.02677)

A recent paper titled "Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filters" proposes using fine-tuned Multimodal Language Models (MLMs) as high-quality filters for image-text data.

Key points:
* Defines multiple metrics to assess image-text quality from different perspectives like object details, text quality, and semantic understanding.
* Leverages GPT-4 and GPT-4V to construct high-quality instruction data for fine-tuning open-source MLMs as effective data filters.
* Fine-tuned MLM filters generate more precise scores, leading to better filtered data and improved performance of pre-trained models on various downstream tasks.

Congrats to the authors for their work!

Paper:  https://huggingface.co/papers/2403.02677
Code: https://github.com/Victorwz/MLM_Filter
Dataset: https://huggingface.co/datasets/weizhiwang/mlm_filter_instructions
Model: https://huggingface.co/weizhiwang/mlm-filter-llava-13b-gpt4v


Introducing the 🤗 Transformers.js WebGPU Embedding Benchmark! ⚡
👉 https://huggingface.co/spaces/Xenova/webgpu-embedding-benchmark 👈

On my device, I was able to achieve a 64.04x speedup over WASM! 🤯 How much does WebGPU speed up ML models running locally in your browser? Try it out and share your results! 🚀

samusenps omaryshchenko thomwolf osanseviero Seuriin Tonic ozzyonfire amiratatomic abidlabs hogunkim felixdrp leekung Noomam xnohat do-me mikestaub

mvaloatto Tonic kramp abidlabs pogzyb cklam12345


OpenCodeInterpreter: Integrating Code Generation with Execution and
  Refinement (2402.14658)

Diaries of Open Source. Part 3! OS goes to the moon!

💻  OpenCodeInterpreter, a family of very powerful code generation models
Models: https://huggingface.co/collections/m-a-p/opencodeinterpreter-65d312f6f88da990a64da456
Paper: https://huggingface.co/papers/2402.14658
Demo https://huggingface.co/spaces/m-a-p/OpenCodeInterpreter_demo

🔷🔶Zephyr 7B Gemma, Gemma fine-tuned with the Zephyr recipe
Model: https://huggingface.co/HuggingFaceH4/zephyr-7b-gemma-v0.1
Demo: https://huggingface.co/spaces/HuggingFaceH4/zephyr-7b-gemma-chat
GH Repo: https://github.com/huggingface/alignment-handbook

🪆The MixedBread folks released a 2D Matryoshka text embedding model, which means you can dynamically change the embedding size and layer counts
Model: https://huggingface.co/mixedbread-ai/mxbai-embed-2d-large-v1
Release blog post: https://www.mixedbread.ai/blog/mxbai-embed-2d-large-v1

🐋Microsoft released Orca Math, which includes 200K grade school math problems
Dataset: https://huggingface.co/datasets/microsoft/orca-math-word-problems-200k

🥷IBM silently released Merlinite, a cool model trained on Mixtral-generated synthetic data using a novel LAB method https://huggingface.co/ibm/merlinite-7b

🌚 Moondream2 - a small vision language model to run on-device!
Model: https://huggingface.co/vikhyatk/moondream2
Demo: https://huggingface.co/spaces/vikhyatk/moondream2

🏙CityDreamer: 3D City Generation
Demo: https://huggingface.co/spaces/hzxie/city-dreamer
Repo: https://github.com/hzxie/city-dreamer
Model: https://huggingface.co/hzxie/city-dreamer

🌏ML in all languages
Sailor, a family of South-East Asian languages models https://huggingface.co/collections/sail/sailor-language-models-65e19a749f978976f1959825
Samvaad dataset, which includes 140k QA pairs in Hindi, Bengali, Marathi, Tamil, Telugu, Oriya, Punjabi, and Gujarati https://huggingface.co/datasets/GenVRadmin/Samvaad-Mixed-Language-2

You can see the previous part at https://huggingface.co/posts/osanseviero/674644082063278

DmitryRyumin fffiloni samusenps vladbogo unclecode visheratin ajibawa-2023 mvaloatto mikonvergence Makya


I have just published my first blog post.

While FlashAttention has been readily integrated into HuggingFace transformers, there are much higher gains to be had (at least theoretically) for finetuning models with examples of variable sequence lengths in a batch.

For a deeper dive, please go through my blog at https://huggingface.co/blog/mayank-mishra/padding-free-transformer.

Kukedlc osanseviero tomaarsen samusenps eramax mayank-mishra julien-c

damerajee julien-c mayank-mishra jeffboudier


GPAvatar: Generalizable and Precise Head Avatar from Image(s) (2401.10215)

🚀🎭🌟 New Research Alert - ICLR 2024! 🌟🎭 🚀
📄 Title: GPAvatar: Generalizable and Precise Head Avatar from Image(s) 🌟🚀

📝 Description: GPAvatar's objective is to faithfully replicate head avatars while providing precise control over expressions and postures.

👥 Authors: Xuangeng Chu et al.

📅 Conference: ICLR, May 7-11, 2024 | Vienna, Austria 🇦🇹

🔗 Paper: https://huggingface.co/papers/2401.10215

🔗 Github Page: https://xg-chu.github.io/project_gpavatar
🔗 Repository: https://github.com/xg-chu/GPAvatar

🔗 Video: https://www.youtube.com/watch?v=7A3DMaB6Zk0

📚 More Papers: more cutting-edge research presented at other conferences in the https://huggingface.co/spaces/DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin

🚀 Added to the Avatars Collection: https://huggingface.co/collections/DmitryRyumin/avatars-65df37cdf81fec13d4dbac36

🔍 Keywords: #GPAvatar #MTA #Synthesis #LipSyncing #Expressions #HighResolutionVideos #ICLR2024 #DeepLearning #Animation #Innovation

DmitryRyumin osanseviero samusenps philipp-zettl rreed-pha


Hi!
We release a series of LLMs named GraphWiz https://huggingface.co/GraphWiz/, which is designed for solving different graph problems and outputs their CoT. GraphWiz has surpassed GPT-4 by a large margin in 9 tasks.

Let us have a try!

Nuo97 osanseviero Kukedlc samusenps kramp Androidkit yoeldcd


Last day on Spaces of the Week ,
and we made it to last place on trending.
i really thought it couldnt get any better, but i'm crying ! 😭

The thing i like the most about ZeroGPU , `import spaces` , is that i dont have to always check to see if someone decided to test if i have hard character limits , and it reloads the application flawlessly .

drop a like on my spaces here :
Spaces of the Week : https://huggingface.co/spaces/tonic/starcoder2
9 other ZeroGPU demos :  https://huggingface.co/tonic


I remember very well that about two years ago, 0-shot named entity recognition (i.e. where you can choose any labels on the fly) was completely infeasible. Fast forward a year, and https://huggingface.co/Universal-NER/UniNER-7B-all surprised me by showing that 0-shot NER is possible! However, I had a bunch of concerns that prevented me from ever adopting it myself. For example, the model was 7B parameters, only worked with 1 custom label at a time, and it had a cc-by-nc-4.0 license.

Since then, a little known research paper introduced GLiNER, which was a modified & finetuned variant of the https://huggingface.co/microsoft/deberta-v3-base line of models. Notably, GLiNER outperforms UniNER-7B, despite being almost 2 orders of magnitude smaller! It also allows for multiple labels at once, supports nested NER, and the models are Apache 2.0.

Very recently, the models were uploaded to Hugging Face, and I was inspired to create a demo for the English model. The demo runs on CPU, and can still very efficiently compute labels with great performance. I'm very impressed at the models.

There are two models right now:
* base (english): https://huggingface.co/urchade/gliner_base
* multi (multilingual): https://huggingface.co/urchade/gliner_multi

And my demo to experiment with the base model can be found here: https://huggingface.co/spaces/tomaarsen/gliner_base

osanseviero MexIvanov urchade rajistics samusenps ajibawa-2023 danielus nickprock polodealvarado deepsh2207 luqman2511 chokamp Poulain Godspeed-AI eek louisguitton


Multi-LoRA Composition for Image Generation (2402.16843)

"Multi-LoRA Composition for Image Generation" introduces two new approaches for combining multiple visual elements in text-to-image generation using Low-Rank Adaptations (LoRAs)! 🎨

Key Points:
* Proposes two methods - LoRA Switch and LoRA Composite - that activate/combine LoRAs during the denoising process rather than merging weights
* LoRA Switch cycles through different LoRAs at each step, while LoRA Composite averages guidance from all LoRAs simultaneously

Paper: https://huggingface.co/papers/2402.16843
Project page: https://maszhongming.github.io/Multi-LoRA-Composition

Congrats to the authors for their work!

DmitryRyumin osanseviero samusenps mvaloatto femboysLover


Introducing https://huggingface.co/datasets/davanstrien/cosmopedia_chat (v0.0.1), my first experiment using the new NousResearch Genstruct model https://huggingface.co/NousResearch/Genstruct-7B

This dataset uses a subset of https://huggingface.co/datasets/HuggingFaceTB/cosmopedia, a synthetic textbook-quality dataset, and Genstruct to generate user/assistant response pairs.

My current results are mixed, but I'm excited to see how much work is happening around synthetic data generation in the community.  Most crucial next step is working more on data filtering from cosmopedia.

Massive thanks to @euclaise @teknium and the other NouseResearch folks for sharing this model ❤


Learning to Generate Instruction Tuning Datasets for Zero-Shot Task
  Adaptation (2402.18334)

Create synthetic instruction datasets using open source LLM's and bonito🐟!

With Bonito, you can generate synthetic datasets for a wide variety of supported tasks.

The Bonito model introduces a novel approach for conditional task generation, transforming unannotated text into task-specific training datasets to facilitate zero-shot adaptation of large language models on specialized data.

This methodology not only improves the adaptability of LLMs to new domains but also showcases the effectiveness of synthetic instruction tuning datasets in achieving substantial performance gains.

AutoBonito🐟: https://colab.research.google.com/drive/1l9zh_VX0X4ylbzpGckCjH5yEflFsLW04?usp=sharing
Original Repo: https://github.com/BatsResearch/bonito?tab=readme-ov-file
Paper: https://huggingface.co/papers/2402.18334

samusenps pabloce clem nihalnayak damerajee jayomb nanyy1025


PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K
  Text-to-Image Generation (2403.04692)

PixArt-Σ

Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation

https://huggingface.co/papers/2403.04692

In this paper, we introduce PixArt-\Sigma, a Diffusion Transformer model~(DiT) capable of directly generating images at 4K resolution. PixArt-\Sigma represents a significant advancement over its predecessor, PixArt-\alpha, offering images of markedly higher fidelity and improved alignment with text prompts. A key feature of PixArt-\Sigma is its training efficiency. Leveraging the foundational pre-training of PixArt-\alpha, it evolves from the `weaker' baseline to a `stronger' model via incorporating higher quality data, a process we term "weak-to-strong training". The advancements in PixArt-\Sigma are twofold: (1) High-Quality Training Data: PixArt-\Sigma incorporates superior-quality image data, paired with more precise and detailed image captions. (2) Efficient Token Compression: we propose a novel attention module within the DiT framework that compresses both keys and values, significantly improving efficiency and facilitating ultra-high-resolution image generation. Thanks to these improvements, PixArt-\Sigma achieves superior image quality and user prompt adherence capabilities with significantly smaller model size (0.6B parameters) than existing text-to-image diffusion models, such as SDXL (2.6B parameters) and SD Cascade (5.1B parameters). Moreover, PixArt-\Sigma's capability to generate 4K images supports the creation of high-resolution posters and wallpapers, efficiently bolstering the production of high-quality visual content in industries such as film and gaming.

samusenps clem mvaloatto victor thomwolf ianyeung DeathGodlike


As promised, a video on the new Major TOM dataset on HF: https://youtu.be/KonWxQ1mCpA?si=tp3Cz6-F0exfgdx3

If you prefer audio, there is a podcast too: https://www.satellite-image-deep-learning.com/p/major-tom-expandable-eo-datasets

I think this is just the start for remote sensing datasets on HuggingFace..!


🗄 Massive data release on the HF Hub for 75 languages!

https://huggingface.co/datasets/BramVanroy/hplt_monolingual_v1_2

In December of last year, HPLT (https://hplt-project.org/) released version 1.2 of their dataset. It covers web-crawled data of 75 languages!, in the raw format as well as deduplicated and cleaned sections. In total, we're talking about over 40TB of data! This data was already accessible via their website but I figured the accessibility could be improved by an integration with Hugging Face tooling. 🤗 So I added the dataset here to the Hugging Face hub, enabing direct use in your conventional training pipelines for LLMs or other language technologies. The data will automatically be downloaded and optimised with just one line of code:

```python
load_dataset("BramVanroy/hplt_mono_v1_2", "nl_cleaned")
```
Let's use this big blob of data to build something awesome in our languages! 🥳

stefan-it samusenps ajibawa-2023 clem osanseviero ymoslem damerajee rafaelpierrehf


We just released bitsandbytes==0.43.0 📦 , with these significant new additions:

‣ 🛫 FSDP+QLoRA support (alpha release)
  ◦ now anyone with 2 powerful gaming GPUs can fine-tune 70B param models at home!
  ◦ in collab with Jeremy Howard + team @ answer.ai
  ◦ answer.ai blogpost: https://www.answer.ai/posts/2024-03-06-fsdp-qlora.html
  ◦ example repo: https://github.com/AnswerDotAI/fsdp_qlora/

‣ 🌈⊞ Official Windows support
  ◦ now via simple `pip install bitsandbytes>=0.43.0`

‣ 📄 Huge docs update:
  ◦ https://huggingface.co/docs/bitsandbytes/main
  ◦ Be sure to check out the optimizers and the API docs
  ◦ ... even more upcoming ...

Under the hood there we have many other improvements, due to extensive maintenance activity, community contributions by super active + knowledgable volunteers ✨ 🚀 and the official sponsorship by Hugging Face that makes all this possible 🤗 ❤ 🌍

We would greatly appreciate any further community contributions, be it to help with refactorings, exterminating flaky tests, writing doc-strings, tutorials, new features. Don't be shy, just contact us and we see where this leads us:
https://github.com/TimDettmers/bitsandbytes/discussions

Have a great weekend everyone!

Titus-von-Koeller mdouglas clem osanseviero rganti ajibawa-2023 julien-c t1u1 ybelkada cstr tolgacangoz hiyouga danielus smangrul

Djruffkutz clem osanseviero samusenps julien-c Theli ybelkada hiyouga smangrul


Speaking of the missing piece in today’s generative AI: Reasoning (or more appropriately, the proper use of Common-Sense)

Human Intelligence is hinged on the brain’s ability to learn vast amounts of background knowledge about the world just by passively observing it. Such common-sense information is believed to be the enabler of intelligent behavior (planning, reasoning and grounding).

Unusual question: how do we actually learn common-sense knowledge?

Unusual opinion: I personally believe we haven’t fully understand how the brain learns, thus cannot get machines to mimic how we learn.

Well so far AI godfather Prof. Yann have quite a promising vision of how machines can learn world models like we humans do. Excited to share his vision after giving I-JEPA a read.

I-JEPA (Image-based Joint-Embedding Predictive Architecture) is a novel approach to self-supervised learning from images. This method focuses on learning semantic image features without relying on pre-set rules based on manual data changes. Instead, I-JEPA predicts the representations of multiple target blocks within a single image using a single context block.

The I-JEPA architecture consists of a context encoder, a target encoder, and a predictor. The context encoder extracts context features from a context block, while the target encoder extracts target features from the target blocks. The predictor then uses the context features to predict the target features.

One of the main advantages of I-JEPA is that it is non-generative, meaning it does not rely on pre-set rules based on manual data changes. It also uses multi-block masking, which allows it to learn semantic representations more effectively.

This is very promising, hopefully we can look back this one day and amuse at how we got it right.

Paper: https://arxiv.org/abs/2301.08243
Code: https://github.com/facebookresearch/ijepa


People in Paris 🇫🇷 🥐

Next week we'll be hosting our first Post-Deployment Data Science Meetup in Paris!

My boss will be talking about Quantifying the Impact of Data Drift on Model
Performance. 👀

The event is completely free, and there's only space for 50 people, so if you are interested, RSVP as soon as possible 🤗

🗓 Thursday, March 14
🕠 5:30 PM - 8:30 PM GMT+1
🔗 RSVP: https://lu.ma/postdeploymentparis


Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis (2401.08503)

🚀🎭🌟 New Research Alert - ICLR 2024! 🌟🎭 🚀
📄 Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis 🌟🚀

👥 Authors: Zhenhui Ye et al.

📅 Conference: ICLR, May 7-11, 2024 | Vienna, Austria 🇦🇹

🔗 Paper: https://huggingface.co/papers/2401.08503

🔗 Github Page: https://real3dportrait.github.io/
🔗 Repository: https://github.com/yerfor/Real3DPortrait

🔥 Model 🤖: https://huggingface.co/ameerazam08/Real3DPortrait

📚 More Papers: more cutting-edge research presented at other conferences in the https://huggingface.co/spaces/DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin

🚀 Added to the Avatars Collection: https://huggingface.co/collections/DmitryRyumin/avatars-65df37cdf81fec13d4dbac36

🔍 Keywords: #Real3D-Potrait #I2P #HTB-SR #A2M #Synthesis #LipSyncing #HighResolutionVideos #ICLR2024 #DeepLearning #Animation #Innovation

DmitryRyumin samusenps clem osanseviero MiSTe-R Saba99

samusenps clem osanseviero DmitryRyumin MiSTe-R


 The past year has seen a rapid pace of delivery of open-to-use Large Language Models (LLMs) for Generative AI. Simultaneously, a number of service providers have bootstrapped, which provide cloud based APIs for generative inference using these LLMs. The list is long: Together.AI, Fireworks.AI, Mosaic ML (Databricks), Anyscale , Perplexity Labs... Assuming that you are fine with privacy and security issues that sending your data to a cloud hosted solution bring, these providers can help launch your Generative AI use case with little friction.

Given that all the cloud based providers provide access to the same/similar open models, how do you distinguish between them? One way is to look at the performance vs cost tradeoff. This post talks about performance.

To understand performance, one needs benchmarks and metrics. There exist a number of benchmarks and leader boards to compare solutions. But, before you start comparing data points flying across the internet, I suggest you to consider the following.

1) Play with your own benchmark: You need to understand the performance for your use case, not some random workload a off-the-shelf benchmark seeks to represent and its authors dreamed off. You need to build and run your own benchmark. You could either understand and re-work the publicly available benchmarks like LLMPerf, or design your own one. Contrary to wisdom, writing a benchmark grounds up isn't hard. I wrote a simple benchmark called GenAI-Bench to compare the response time of cloud providers. You can do it too.
2) Understand the metrics you care about. Is your use case a chatbot which needs a quick turnaround for the first word? Or do you need to parse a large number of documents and get a summary over-night. Correctly identifying what you care for will help you narrow down the best provider for your needs.
3) Keep generating data points periodically. Cloud providers routinely update their software. Metrics change. Measure them periodically.


After the Supervised Fine-Tuning (SFT) phase, we observed a notable degradation in the instruction-following capabilities of the LLaVA Multi-Modal Large Language Model (MM-LLM). To address this issue, we introduced a 6K-entry VQA preference dataset and employed Direct Preference Optimization (DPO), alongside testing other algorithms such as Rejection Sampling and SteerLM, to enhance instruction-following proficiency. Our methodology not only fully restored the language following capabilities of LLaVa on the MT-Bench but also outperformed LLaVA-RLHF and Vicuna. Additionally, our approach extended to visual VQA tasks, as demonstrated by significant performance improvements on MM-Vet and LLaVa-Bench. An interesting observation was that, compared to models using distilled SFT, our method showed substantial out-of-distribution improvements.

https://arxiv.org/abs/2402.10884
Model available
https://huggingface.co/alexshengzhili/llava-v1.5-13b-dpo
GitHub:
https://github.com/findalexli/mllm-dpo/edit/main/README.MD


Dear music lovers 🕺,

MusicLang Space is now live: https://huggingface.co/spaces/musiclang/README

MusicLang is a controllable model for music generation:

> 🦙 Discover the LLAMA2 architecture, trained from scratch for symbolic music generation, ensuring exceptional quality;
> 👨🎨 Unleash your creativity by extending an existing music, or create new ones from scratch;
> 🤖 Integrate MusicLang into your applications, with an inference optimized for CPUs written in C, other integrations and optimizations coming soon.

In the space, you’ll find :

1⃣   MusicLang foundation model: our fondation model for creating and generating original midi soundtracks https://huggingface.co/musiclang/musiclang-v2;

2⃣  MusicLang predict: our AI prediction api of the MusicLang package https://github.com/musiclang/musiclang_predict?tab=readme-ov-file;

3⃣  MusicLang Language:a new language for tonal music. This language allows composers to load, write, transform and predict symbolic music in a simple, condensed and high level manner  https://github.com/MusicLang/musiclang;

4⃣ MusicLang Demo Space: https://huggingface.co/spaces/musiclang/musiclang-predict

5⃣ Our Colab: https://colab.research.google.com/drive/1MA2mek826c05BjbWk2nRkVv2rW7kIU_S?usp=sharing

Help us share the future of music composition! Spread the word, show your support by adding a star or contribute to our project. ⭐✨

Music Sounds Definitely Better with You 🎶 🖤

cc @floriangardin @MehdiLeZ @reach-vb

Thanks a lot,

The MusicLang team ❤

floriangardin reach-vb samusenps Bils kramp fffiloni clem osanseviero victor MehdiLeZ asigalov61 RadMann


GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (2403.03507)

GaLore

Memory-Efficient LLM Training by Gradient Low-Rank Projection

https://huggingface.co/papers/2403.03507

Training Large Language Models (LLMs) presents significant memory challenges, predominantly due to the growing size of weights and optimizer states. Common memory-reduction approaches, such as low-rank adaptation (LoRA), add a trainable low-rank matrix to the frozen pre-trained weight in each layer, reducing trainable parameters and optimizer states. However, such approaches typically underperform training with full-rank weights in both pre-training and fine-tuning stages since they limit the parameter search to a low-rank subspace and alter the training dynamics, and further, may require full-rank warm start. In this work, we propose Gradient Low-Rank Projection (GaLore), a training strategy that allows full-parameter learning but is more memory-efficient than common low-rank adaptation methods such as LoRA. Our approach reduces memory usage by up to 65.5% in optimizer states while maintaining both efficiency and performance for pre-training on LLaMA 1B and 7B architectures with C4 dataset with up to 19.7B tokens, and on fine-tuning RoBERTa on GLUE tasks. Our 8-bit GaLore further reduces optimizer memory by up to 82.5% and total training memory by 63.3%, compared to a BF16 baseline. Notably, we demonstrate, for the first time, the feasibility of pre-training a 7B model on consumer GPUs with 24GB memory (e.g., NVIDIA RTX 4090) without model parallel, checkpointing, or offloading strategies.

osanseviero notsahil andysalerno nofreewill femboysLover OneLoneTurnip boapps arcdyn megataro bytesizedllm clem jthk-HF rganti shing3232 mrm8488


BioT5: Enriching Cross-modal Integration in Biology with Chemical
  Knowledge and Natural Language Associations (2310.07276)

BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations

BioT5 achieves superior performance on various biological tasks by integrating natural language. See more details at:

Paper: https://huggingface.co/papers/2310.07276
Code: https://github.com/QizhiPei/BioT5
Model (base): https://huggingface.co/QizhiPei/biot5-base
Model (molecule captioning): https://huggingface.co/QizhiPei/biot5-base-mol2text
Model (Text-based Molecule Design): https://huggingface.co/QizhiPei/biot5-base-text2mol

osanseviero victor katielink samusenps Tonic QizhiPei clem


Extreme Compression of Large Language Models via Additive Quantization (2401.06118)

QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models (2310.16795)

Check out quantized weights from ISTA-DAS Lab directly in their organisation page: https://huggingface.co/ISTA-DASLab ! With official weights of AQLM (for 2bit quantization) & QMoE (1-bit MoE quantization)

Read more about these techniques below:

AQLM paper: https://hf.co/papers/2401.06118
QMoE: https://huggingface.co/papers/2310.16795

Some useful links below:

AQLM repo: https://github.com/Vahe1994/AQLM
How to use AQLM & transformers: https://huggingface.co/docs/transformers/quantization#aqlm
How to use AQLM & PEFT: https://huggingface.co/docs/peft/developer_guides/quantization#aqlm-quantizaion

Great work from @BlackSamorez and team !

alielfilali01 marcsun13 osanseviero samusenps FM-1976 mayank-mishra clem goodgame dillfrescott


🚀 Major Update: OpenLLM Turkish Benchmarks & Leaderboard Launch! 🚀

Exciting news for the Hugging Face community! I'm thrilled to announce the launch of my fully translated OpenLLM Benchmarks in Turkish, accompanied by my innovative leaderboard, ready to highlight the capabilities of Turkish language models. This marks a landmark achievement in supporting and advancing Turkish AI research.

What’s New:

📚 Complete OpenLLM Benchmarks in Turkish: Dive into my comprehensive suite of benchmarks, now available for thorough evaluation of Turkish LLMs.

📈 Live Leaderboard: Explore my live leaderboard showcasing the progress and excellence in Turkish language AI. (Note: Current evaluations are conducted manually but are consistently updated.)

Partnership Invitation:

🤝 Join My Automation Mission: I'm on the lookout for partners to help transition from manual to automated leaderboard evaluations. Your support can catalyze real-time, streamlined assessments, pushing Turkish LLMs to new heights.
Key Resources:

📚 Explore the Turkish OpenLLM Collection: (https://huggingface.co/collections/malhajar/openllmturkishleadboard-datasets-65e5854490a87c0f2670ec18)

🏆 Discover the Leaderboard: (https://huggingface.co/spaces/malhajar/OpenLLMTurkishLeaderboard)

Get Involved:

💡 Share Your Models: Contribute to the burgeoning field of Turkish AI, showcasing your work and contributing to the collective progress.

Let's unite to propel Turkish AI forward and set a precedent for the global community. Stay tuned as I plan to expand these efforts to other languages, further enriching the AI ecosystem!

Join this groundbreaking endeavor and let’s shape the future of AI together! 🌐

#TurkishLLM #AI #MachineLearning #LanguageModels #OpenLLM #HuggingFace

badayvedat osanseviero clefourrier Wauplin uisikdag samusenps comarproject thomwolf clem Yhyu13 ahmet27 mohammedbriman

kyo-takano victor Wauplin comarproject thomwolf HamzaMuaz ahmet27


`mamba` is now available in transformers.  Thanks to @tridao and @albertgu  for this brilliant model! 🚀 and the amazing `mamba-ssm` kernels powering this!
Checkout the collection here:
https://huggingface.co/collections/state-spaces/transformers-compatible-mamba-65e7b40ab87e5297e45ae406

lysandre osanseviero susnato victor ZetangForward samusenps chansung tomaarsen ceyda zjr2000


Anchor Large Language Models: Up to 99% KV cache reduction!

paper: https://arxiv.org/pdf/2402.07616.pdf

osanseviero damerajee samusenps pangjh3 clem Excursions Tonic


LLM “Patchnization”

Minimal, clean code for video/image "patchnization" - a process commonly used in tokenizing visual data for use in a Transformer encoder.

Code: https://github.com/Jaykef/min-patchnizer

The code above, first extracts still images (frames) from a video, splits the image frames into smaller fixed-size patches, linearly embeds each of them, adds position embeddings and then saves the resulting sequence of vectors for use in a Vision Transformer encoder. I tried training the resulting sequence vectors with Karpathy's minbpe and it took ~2173s per frame to tokenize. The whole "patchnization" took ~77.4s for a 20s video on my M2 Air.

The files in the repo work as follows:

1. patchnizer.py: Holds code for simple implementation of the three stages involved (extract_image_frames from video, reduce image_frames_to_patches of fixed sizes 16x16 pixels, then linearly_embed_patches into a 1D vector sequence with additional position embeddings.

2. patchnize.py: performs the whole process with custom configs (patch_size, created dirs, video - I am using the "dogs playing in snow" video by sora).

3. train.py: Trains the resulting one-dimensional vector sequence (linear_patch_embeddings + position_embeddings) on Karpathy's minbpe (a minimal implementation of the byte-pair encoding algorithm).

4. check.py: Checks if the patch embeddings match the original image patches by recovering the image frames from their corresponding image patches.

The Patchnizer class has three stubs:
- extract_image_frames() which chops the video (20sec) into 60 frames (i.e each frame is ~0.33 secs) each of size 1280x720 pixels (original video dims).
- image_frames_to_patches() which grids each image frame into 16x16 pixels tiles. This makes each frame has a total of 3600 image patches (i.e 80 rows by 45 columns).
- linearly_embed_patches() turns the image patches into patch embeddings (a long string of integers for each image patch) then adds a position embedding for each patch.

osanseviero femboysLover samusenps clem Lwasinam whooray cahlen


Stopwolf julien-c samusenps clem dillfrescott


Released updated weights for moondream2 today, with significantly improved benchmark scores!

https://huggingface.co/spaces/vikhyatk/moondream2

https://huggingface.co/vikhyatk/moondream2

Sylvestre osanseviero victor dim samusenps Dlbk clem Lewdiculous Tom-Neverwinter not-lain MoonRide


🚀 Introducing UltraTextbooks v2: The Ultimate Educational NLP Dataset! 📚

I've expanded the dataset to include an even wider range of high-quality textbooks, with a special focus on machine learning, mathematics, and coding. 💻🧮

With over 3 million examples and 6 GB of data, UltraTextbooks v2 is your go-to resource for training advanced language models and developing cutting-edge educational applications. 🎓

Explore the dataset on Hugging Face and unlock the power of AI in education! 🔓

https://huggingface.co/datasets/Locutusque/UltraTextbooks-2.0

osanseviero Felladrin clem samusenps Jjjjjo


Empower yourself with knowledge and skills. The power of knowledge is the key for advancement. Realize that without discovering your weaknesses, it would be hard to improve yourself. No one is perfect, no one knows everything, but we all have the freedom to learn and discover our true potential. It's important to surround yourself with positivity and people who help you push forward to improve. Lifelong learning is a crucial key of success.

#LifelongLearning #Success #Enthusiasm


Hi everyone,

I'd like to share our free GPT-4 chatbot: https://huggingface.co/spaces/yuntian-deng/ChatGPT4. Data collected from it will be shared back with the community in future releases of the WildChat dataset: https://huggingface.co/datasets/allenai/WildChat. Please help us reach 1 million conversations!

Thanks,
Yuntian

yuntian-deng iDrops jinghan23 ajibawa-2023 osanseviero samusenps Dlbk Tonic Lewdiculous psy-taha artghoul RobertRoss asp933 IdleIdiot SwedMlite Maria200035

artghoul Aerialyn lololololololokok caiquemoa Johanbb


Design2Code: How Far Are We From Automating Front-End Engineering? (2403.03163)

The "Design2Code: How Far Are We From Automating Front-End Engineering" paper presents a benchmark for multimodal large language models (LLMs) aimed at automating front-end web development by translating webpage designs (screenshots) into code. This task evaluates the models' ability to recreate webpages that are visually and structurally similar to the original designs.

Key Points:
* Introduces the Design2Code task and benchmark for converting webpage screenshots into code, aiming to automate front-end web development.
* Evaluates multimodal LLMs using comprehensive metrics for visual similarity and element matching.
* GPT-4V outperforms other models in terms of visual resemblance and content accuracy, with generated webpages often preferred over the original references.

Paper: https://huggingface.co/papers/2403.03163
Project page: https://salt-nlp.github.io/Design2Code/
Dataset: https://huggingface.co/datasets/SALT-NLP/Design2Code

Congrats to the authors for their work!


Learning a Decision Tree Algorithm with Transformers (2402.03774)

huggingface? X(formerly known as twitter)!

Yufan here, CS PhD student from UC San Diego.

LLMs as ML algorithms? see my latest work MetaTree: https://huggingface.co/papers/2402.03774

Read more:

https://evanzhuang.github.io/
https://twitter.com/yufan_zhuang

yzhuang qnguyen3 osanseviero victor samusenps shing3232 ZennyKenny


Wow! Hello world. How's ECCV submissions going?


Hi everyone!

Looking forward to engaging with you all 🤗


radames DmitryRyumin weizhey osanseviero stefan-it apol taesiri ZennyKenny den0620 samusenps edeani rahmanansari iv7dev Nacholmo erickdp EddyGiusepe jbilcke-hf leegao19 Noomam

rtscott2001 edeani CreitinGameplays jbilcke-hf


Sharing our paper and library for building LLM agent. The library is less than 1K code lines!
https://github.com/SalesforceAIResearch/AgentLite
https://arxiv.org/abs/2402.15538

clem osanseviero taesiri qgao007 victor samusenps kumarh1982 Hanyu66 theArif


osanseviero Suparious samusenps dashfunnydashdash ZennyKenny thomwolf Lewdiculous victor


👋 I'm Sam

I use ML and HPC to accelerate scientific discovery @ Argonne National Laboratory*

https://samforeman.me
https://twitter.com/saforem2
https://github.com/saforem2

* https://alcf.anl.gov/about/people/sam-foreman

osanseviero tomaarsen victor medmac01 mozayed


weizhey osanseviero tomaarsen MarinaraSpaghetti samusenps polodealvarado Lewdiculous HaiderSultanArc


Hi everyone,

I'd like to share our project on open-type Named Entity Recognition (NER). Our model uses a transformer encoder (BERT-like), making the computation overhead very minimal compared to use of LLMs. I've developed a demo that runs on CPU on Google Colab.

Colab Demo: https://colab.research.google.com/drive/1mhalKWzmfSTqMnR0wQBZvt9-ktTsATHB?usp=sharing

Code: https://github.com/urchade/GLiNER

Paper: https://arxiv.org/abs/2311.08526

clem NePe sgarbi osanseviero samusenps Tanvir1337 medmac01 merve ruman1 awinml maortal Antoine imihalcea richgjacksonaz Bailefan eek


DeepLearning.AI just announced a new short course: Open Source Models with Hugging Face 🤗, taught by Hugging Face's own Maria Khalusova, Marc Sun and Younes Belkada!

As many of you already know, Hugging Face has been a game changer by letting developers quickly grab any of hundreds of thousands of already-trained open source models to assemble into new applications. This course teaches you best practices for building this way, including how to search and choose among models.

You'll learn to use the Transformers library and walk through multiple models for text, audio, and image processing, including zero-shot image segmentation, zero-shot audio classification, and speech recognition. You'll also learn to use multimodal models for visual question answering, image search, and image captioning. Finally, you’ll learn how to demo what you build locally, on the cloud, or via an API using Gradio and Hugging Face Spaces.

Thank you very much to Hugging Face's wonderful team for working with us on this.

You can sign up for the course here: https://www.deeplearning.ai/short-courses/open-source-models-hugging-face/

osanseviero MariaK clem abidlabs Wauplin thomwolf Jofthomas kramp julien-c Violette philschmid VictorSanh lhoestq victor DmitryRyumin brunatrevelin m-ric loubnabnl samusenps leegao19 radames P3rry JAkinyemi qgao007 mvaloatto jeffboudier ybelkada hysts neerajjulka lysandre linoyts muhtasham madroid marcsun13 keerekeerweere OfferL MarinaraSpaghetti bulentsiyah dijvar pankaj-munde medmac01 beenakurian

clem abidlabs Wauplin thomwolf osanseviero Jofthomas julien-c Violette philschmid lhoestq victor katielink loubnabnl samusenps rowbradley hmb andrewrreed radames rtscott2001 yjernite jeffboudier ybelkada lysandre muhtasham madroid marcsun13 quibblerquery dimbyTa mayacinka

osanseviero clem abidlabs Wauplin thomwolf Jofthomas julien-c Violette philschmid VictorSanh lhoestq victor brunatrevelin m-ric loubnabnl radames ybelkada lysandre muhtasham marcsun13


Design2Code

How Far Are We From Automating Front-End Engineering?

https://huggingface.co/papers/2403.03163

Generative AI has made rapid advancements in recent years, achieving unprecedented capabilities in multimodal understanding and code generation. This can enable a new paradigm of front-end development, in which multimodal LLMs might directly convert visual designs into code implementations. In this work, we formalize this as a Design2Code task and conduct comprehensive benchmarking. Specifically, we manually curate a benchmark of 484 diverse real-world webpages as test cases and develop a set of automatic evaluation metrics to assess how well current multimodal LLMs can generate the code implementations that directly render into the given reference webpages, given the screenshots as input. We also complement automatic metrics with comprehensive human evaluations. We develop a suite of multimodal prompting methods and show their effectiveness on GPT-4V and Gemini Pro Vision. We further finetune an open-source Design2Code-18B model that successfully matches the performance of Gemini Pro Vision. Both human evaluation and automatic metrics show that GPT-4V performs the best on this task compared to other models. Moreover, annotators think GPT-4V generated webpages can replace the original reference webpages in 49% of cases in terms of visual appearance and content; and perhaps surprisingly, in 64% of cases GPT-4V generated webpages are considered better than the original reference webpages. Our fine-grained break-down metrics indicate that open-source models mostly lag in recalling visual elements from the input webpages and in generating correct layout designs, while aspects like text content and coloring can be drastically improved with proper finetuning.

victor DavidVivancos hmb vladbogo clem TheSaifurRahman sgarbi krishnapraveen samusenps jnopareboateng seanmiranda julien-c awacke1 marcelovicentegc Ananze


Hey, it took some time but I finally moved out and got internet back, so here I am again!
A lot of things to get updated on, I will try to reply to each of you ASAP.
See you soon!

kawaiibastard RUcrittingME Ainonake MarinaraSpaghetti victor okeksama papercanteen111 saishf WaefreBeorn eevee32x antiven0m 010O11 damerajee CarlosIABn samusenps AtonMountlook krisshen Zeldazackman Diavator Adriato wath5 ConquestofElysium jarjarbinks SGNghiax SerialKicked ClaudioItaly WilsonWan IHaBiS locke12 WeForgot Shadowplague Tillx85

mayurninama bluuwhale ZZnake rywiz Diavator cpham9000 xpgx1 AtonMountlook frgthzjmk


hey there folks new Yi model just came out , and i had a gradio interface ready since their last releases.

it's just a base model but you can check it out here : https://huggingface.co/spaces/Tonic/Yi-9B

cant wait to fine tune this one 🤗🚀


You can now use gpt4all.io to instantly search, download, and chat with models hosted on huggingface!

osanseviero victor diegotluz samusenps ybelkada Tonic pcuenq ajibawa-2023 davanstrien clem zpn EtienneDosSantos Nymbo

osanseviero samusenps ybelkada alielfilali01 pcuenq clem Hugorowan zpn PeepDaSlan9 Nymbo


SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis (2311.17590)

🚀😈🌟 New Research Alert - CVPR 2024! 🌟😈 🚀
📄 Title: SyncTalk: The Devil 😈 is in the Synchronization for Talking Head Synthesis 🌟🚀

📝 Description: SyncTalk synthesizes synchronized talking head videos, employing tri-plane hash representations to maintain subject identity. It can generate synchronized lip movements, facial expressions, and stable head poses, and restores hair details to create high-resolution videos.

👥 Authors: Ziqiao Peng et al.

📅 Conference: CVPR, Jun 17-21, 2024 | Seattle WA, USA 🇺🇸

🔗 Paper: https://huggingface.co/papers/2311.17590

🔗 Github Page: https://ziqiaopeng.github.io/synctalk
🔗 Repository: https://github.com/ZiqiaoPeng/SyncTalk

📚 More Papers: more cutting-edge research presented at other conferences in the https://huggingface.co/spaces/DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin

🚀 Added to the Avatars Collection: https://huggingface.co/collections/DmitryRyumin/avatars-65df37cdf81fec13d4dbac36

🔍 Keywords: #TalkingHeads #Synthesis #TriPlaneHash #FacialExpressions #LipSyncing #HighResolutionVideos #CVPR2024 #DeepLearning #Animation #Innovation

DmitryRyumin osanseviero samusenps minhdang Tonic takeraparterer brainhome MiSTe-R JasonZJ victor vladbogo clem dashfunnydashdash lvalue


Diaries of Open Source. Part 2. Open Source is going brrrrr

🚀The European Space Agency releases MajorTOM, a dataset of earth observation covering half the earth. The dataset has 2.5 trillion pixels! Congrats @aliFrancis and @mikonvergence !
Dataset: https://huggingface.co/datasets/Major-TOM/Core-S2L2A
Viewer: https://huggingface.co/spaces/Major-TOM/MajorTOM-Core-Viewer

🍞Re-ranking models by MixedBreadAI, with very high quality, Apache 2 license, and easy to use!
Models: https://huggingface.co/models?other=reranker&sort=trending&search=mixedbread-ai
Blog: https://www.mixedbread.ai/blog/mxbai-rerank-v1

🧊StabilityAI and TripoAI release TripoSR, a super-fast MIT-licensed image-to-3D model!
Model: https://huggingface.co/stabilityai/TripoSR
Demo: https://huggingface.co/spaces/stabilityai/TripoSR

🤝Together AI and HazyResearch release Based
Models and datasets: https://huggingface.co/collections/hazyresearch/based-65d77fb76f9c813c8b94339c
GH repo: https://github.com/HazyResearch/based

🌊LaVague: an open-source pipeline to turn natural language into browser actions! It can run locally with https://huggingface.co/HuggingFaceH4/zephyr-7b-gemma-v0.1
Read more about it at https://huggingface.co/posts/dhuynh95/717319217106504

🏆Berkeley Function-Calling Leaderboard
Read about it: https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html
Leaderboard: https://gorilla.cs.berkeley.edu/leaderboard.html

🐬Sailor-Chat: chat models built on top of OpenOrca and @sarahooker CohereForAI Aya project. They can be used for South-East Asia languages such as Indonesian, Thai, Vietnamese, Malay and Lao!
Models: https://huggingface.co/collections/sail/sailor-language-models-65e19a749f978976f1959825
Demo: https://huggingface.co/spaces/sail/Sailor-7B-Chat

🤗Arabic-OpenHermes-2.5: OpenHermes dataset translated to Arabic https://huggingface.co/datasets/2A2I/Arabic-OpenHermes-2.5

See the previous part here https://huggingface.co/posts/osanseviero/622788932781684

mrm8488 mikonvergence peterschmidt85 DmitryRyumin yjernite SivilTaram zirui3 ajibawa-2023 davanstrien vladbogo clem irenesolaiman mvaloatto Jose7juanFdz kramp juliuslipp

kshitizkhanal7 samusenps SivilTaram clefourrier lucabaggi clem irenesolaiman nisten