Sailor: Open Language Models for South-East Asia (2404.03608)

✨ Today, we're excited to share the full data processing script used in developing our Sailor models. The repo provides an end-to-end data processing pipeline for LLM training. 🚀

💻Code: https://github.com/sail-sg/sailcraft
🤗Model: https://huggingface.co/collections/sail/sailor-language-models-65e19a749f978976f1959825
📜Paper: https://huggingface.co/papers/2404.03608
🌐Homepage: https://sailorllm.github.io

# Overview 🔍

The pipeline consists of 4 stages🧹:
1⃣ Initial data cleaning
2⃣ Near deduplication
3⃣ Exact deduplication
4⃣ Second round of data cleaning

A special focus was given to the data cleaning part of South-East Asian (SEA) languages🌍

# Use Case ✨

With this codebase, you can clean your own dataset with:

✅ Get filtered data counts after each processing stage
✅ Easily configure language-specific cleaning rules (we support Arabic, Bengali, Catalan, Spanish, Basque, French, Hindi, Portuguese, Urdu, and optimize for English, Indonesian, Vietnamese, Chinese, Thai, Lao, Malay)
✅ Investigate what data was removed at each processing stage

# Acknowledgement 🙏

The main credit goes to @dreamerdeo , the first author of our Sailor paper ❤! He put in tremendous effort on the data processing pipeline, enabling the model's great performance. We believe the mini repo will be a valuable resource for researchers working on dataset curation for large language models. 🎉

Sharing the recipe openly aligns with our commitment to open language model development. 💪 And this repo would not have been possible without the contributions from the open community, including the BigScience data cleaning tool, the all-in-one deduplication tool by @chenghao , and the deduplication project from Google. 🧠

# What's Next 🚀

Share your thoughts or leave any comments on what you'd like the Sailor models to do! We also have some exciting news coming soon, and please stay tuned. 🚄

mmhamdy dreamerdeo ToluClassics chenghao


#Previous Version / Older
📂Huggingface for Android ➡
🪶Median ( Go Native ) Plugin :

version 0.0.1
🚀 https://huggingface.co/spaces/prithivMLmods/Huggingface-Android-App

jeanflop samusenps Taylor658 ucalyptus raincandy-u AlekseiPravdin dillfrescott pedi Ramikan-BR prithivMLmods


A Careful Examination of Large Language Model Performance on Grade
  School Arithmetic (2405.00332)

A Careful Examination of Large Language Model Performance on Grade School Arithmetic

https://huggingface.co/papers/2405.00332

Large language models (LLMs) have achieved impressive success on many benchmarks for mathematical reasoning. However, there is growing concern that some of this performance actually reflects dataset contamination, where data closely resembling benchmark questions leaks into the training data, instead of true reasoning ability. To investigate this claim rigorously, we commission Grade School Math 1000 (GSM1k). GSM1k is designed to mirror the style and complexity of the established GSM8k benchmark, the gold standard for measuring elementary mathematical reasoning. We ensure that the two benchmarks are comparable across important metrics such as human solve rates, number of steps in solution, answer magnitude, and more. When evaluating leading open- and closed-source LLMs on GSM1k, we observe accuracy drops of up to 13%, with several families of models (e.g., Phi and Mistral) showing evidence of systematic overfitting across almost all model sizes. At the same time, many models, especially those on the frontier, (e.g., Gemini/GPT/Claude) show minimal signs of overfitting. Further analysis suggests a positive relationship (Spearman's r^2=0.32) between a model's probability of generating an example from GSM8k and its performance gap between GSM8k and GSM1k, suggesting that many models may have partially memorized GSM8k.


🚀 Just released version 0.23.0 of the huggingface_hub Python library!

Exciting updates include:
📁 Seamless download to local dir!
💡 Grammar and Tools in InferenceClient!
🌐  Documentation full translated to Korean!
👥 User API: get likes, upvotes, nb of repos, etc.!
🧩 Better model cards and encoding for ModelHubMixin!

Check out the full release notes for more details:
https://huggingface.co/spaces/Wauplin/huggingface_hub/discussions/6
 👀

severo tomaarsen Lewdiculous julien-c albertvillanova mmhamdy radames davanstrien Molbap not-lain


@VanshGehlot could upgrade help Me with this


Full parameter fine-tuning of the LLaMA-3 8B model using a single GTX 3090 GPU with 24GB of graphics memory?

Please check out our tool for fine-tuning, inferencing, and evaluating GreenBitAI's low-bit LLMs:
https://github.com/GreenBitAI/green-bit-llm
Model Zoo:
https://huggingface.co/GreenBitAI


MotionLCM: Real-time Controllable Motion Generation via Latent
  Consistency Model (2404.19759)

🔥Motion Latent Consistency Model🔥

Introducing MotionLCM💃, controlling and generating a motion in milliseconds!

Huggingface Space:
https://huggingface.co/spaces/wxDai/MotionLCM
Huggingface Paper:
https://huggingface.co/papers/2404.19759

Project page: https://dai-wenxun.github.io/MotionLCM-page/
Paper: https://arxiv.org/pdf/2404.19759.pdf
Code: https://github.com/Dai-Wenxun/MotionLCM
video: https://www.youtube.com/watch?v=BhrGmJYaRE4

MotionLCM supports inference pipelines of 1-4 steps, with almost no difference in effectiveness between 1 and 4 steps. Generating approximately 200 frames of motion only takes about 30ms, which averages to approximately 6k fps per frame.

Our MotionLCM can achieve high-quality text-to-motion and precise motion control results (both sparse and dense conditions) in ∼30 ms.

We integrated a control module into the diffusion of the latent space, named Motion ControlNet, to achieve controllable motion generation. Our control algorithm is approximately 1,000 times faster than the best-performing baseline, with comparable quality.


Custom pipelines and components in Diffusers 🎸

Wanted to use customized pipelines and other components (schedulers, unets, text encoders, etc.) in Diffusers?

Found it inflexible?

Since the first dawn on earth, we have supported loading custom pipelines via a `custom_pipeline` argument 🌄

These pipelines are inference-only, i.e., the assumption is that we're leveraging an existing checkpoint (e.g., runwayml/stable-diffusion-v1-5) and ONLY modifying the pipeline implementation.

We have many cool pipelines, implemented that way. They all share the same benefits available to a `DiffusionPipeline`, no compromise there 🤗

Check them here:
https://github.com/huggingface/diffusers/tree/main/examples/community

Then we might have a requirement of everything customized i.e., custom components along with a custom pipeline. Sure, that's all possible.

All you have to do is keep the implementations of those custom components on the Hub repository you're loading your pipeline checkpoint from.

SDXL Japanese was implemented like this 🔥
https://huggingface.co/stabilityai/japanese-stable-diffusion-xl

Full guide is available here ⬇
https://huggingface.co/docs/diffusers/main/en/using-diffusers/custom_pipeline_overview

And, of course, these share all the benefits that come with `DiffusionPipeline`.


When I read the KAN paper, I see physicists casually making fun of the uncertainties in MLPs or Neural nets as a whole:

- "The philosophy here is close to the mindset of physicists, who often care more about typical cases rather than worst cases" lol this went hard on NNs

- "Finite grid size can approximate the function well with a residue rate independent of the dimension, hence beating curse of dimensionality!" haha.

- "Neural scaling laws are the phenomenon where test loss decreases with more model parameters"

- "Our approach, which assumes the existence of smooth Kolmogorov Arnold representations, decomposes the high-dimensional function into several 1D functions"

Key Differences With MLPs:
- Activation Functions: Unlike MLPs that use fixed activation functions at the nodes, KANs utilize learnable activation functions located on the edges between nodes.
- Weight Parameters: In KANs, traditional linear weight matrices are absent. Instead, each weight parameter is replaced by a learnable univariate function, specifically a spline.
- Summation Nodes: Nodes in KANs perform simple summation of incoming signals without applying non-linear transformations.

Advantages Over MLPs:
- Accuracy: achieve higher accuracy with smaller network sizes compared to larger MLPs in tasks like data fitting and solving partial differential equations (PDEs).
- Interpretability: Due to their unique structure, KANs are more interpretable than MLPs.

Technical Innovations:
- Learnable Edges: learnable functions on network edges presents a novel approach to network design, providing greater flexibility in modeling complex relationships in data.
- No Linear Weights: elimination of linear weights reduces the parameters, and potentially simplifies the learning process, focusing on the optimization of univariate function representations.

Applications and Practical Use:
- Scientific Collaboration: KANs have been applied in scientific settings as tools to help discover or rediscover math


TGI v2.0.2 is out!
- New models (idefics2, phi3)
- Cleaner VLM support in the openai layer
- Upgraded to pytorch 2.3.0

https://github.com/huggingface/text-generation-inference/releases/tag/v2.0.2

Kudos @Narsil @olivierdehaene @drbh and so many contributors!

sosoai Narsil VictorSanh tomaarsen radames Alagib dawengmerak18 clboetticher


KAN: Kolmogorov-Arnold Networks (2404.19756)

Okay, first pass over KAN: Kolmogorov–Arnold Networks, it looks very interesting!

Interpretability of KAN model:
May be considered mostly as a safety issue these days, but it can also be used as a form of interaction between the user and a model, as this paper argues and I think they make a valid point here. With MLP, we only interact with the outputs, but KAN is an entirely different paradigm and I find it compelling.

Scalability:
KAN shows better parameter efficiency than MLP. This likely translates also to needing less data. We're already at the point with the frontier LLMs where all the data available from the internet is used + more is made synthetically...so we kind of need something better.

Continual learning:
KAN can handle new input information w/o catastrophic forgetting, which helps to keep a model up to date without relying on some database or retraining.

Sequential data:
This is probably what most people are curious about right now, and KANs are not shown to work with sequential data yet and it's unclear what the best approach might be to make it work well both in training and regarding the interpretability aspect. That said, there's a rich long history of achieving sequential data in variety of ways, so I don't think getting the ball rolling here would be too challenging.

Mostly, I just love a new paradigm and I want to see more!

https://huggingface.co/papers/2404.19756

valeriylo victor mmhamdy radames turiabu jorostuff KillerShoaib jgitsolutions dillfrescott Roozbehr72 Adien123 makiisthebes fendy07

Jaward sergeodesico rennokki sseres Noomam dillfrescott Pwlot nebiyu29


🪽 New FluentlyXL V4!

🤗 Finally the model came out, it turned out great, we worked on contrast, realism and anatomy.

👨💻 Model page:  https://huggingface.co/fluently/Fluently-XL-v4

🦾 You can try it here: https://huggingface.co/spaces/fluently/Fluently-Playground

ehristoforu dreamdrop-art lunarflu victor radames DmitryRyumin Delik frankmao lailailailailai

victor ehristoforu radames dreamdrop-art


Can you create domain-specific synthetic datasets in under 20 minutes?

@burtenshaw recently launched the Domain Specific Dataset Project as part of Data is Better Together. As part of this, Ben created a Space that you can use to define some key perspectives and concepts from a domain. This seed dataset can then be used to generate a synthetic dataset for a particular domain.

In less than 30 minutes this afternoon, I created a domain-specific dataset focused on data-centric machine learning using these tools: https://huggingface.co/datasets/davanstrien/data-centric-ml-sft.

You can create your own domain specific datasets using this approach. Find the steps to follow here: https://github.com/huggingface/data-is-better-together/blob/main/domain-specific-datasets/README.md

alvarobartt lunarflu ZennyKenny fdaudens burtenshaw tomaarsen Taylor658 ucyang


If you are a debate fan or did this as an extracurricular activity as a kid, you might have fun with this demo - debate bot. Debate against AI/RAG:

https://huggingface.co/spaces/vectara/debate-bot


New conversation in our Journalists on Hugging Face community: Exploring auto-tagging articles for taxonomy.

I've shared insights from my previous experience with fine-tuning a model for a classification task. But has anyone built a similar use case? Or are you seeking a solution for this task too? Join the discussion here: https://huggingface.co/spaces/JournalistsonHF/README/discussions/2#6631b9a250c203b5a925d2e1


Octopus v4: Graph of language models (2404.19296)

Octopus v4

Graph of language models

https://huggingface.co/papers/2404.19296

Language models have been effective in a wide range of applications, yet the most sophisticated models are often proprietary. For example, GPT-4 by OpenAI and various models by Anthropic are expensive and consume substantial energy. In contrast, the open-source community has produced competitive models, like Llama3. Furthermore, niche-specific smaller language models, such as those tailored for legal, medical or financial tasks, have outperformed their proprietary counterparts. This paper introduces a novel approach that employs functional tokens to integrate multiple open-source models, each optimized for particular tasks. Our newly developed Octopus v4 model leverages functional tokens to intelligently direct user queries to the most appropriate vertical model and reformat the query to achieve the best performance. Octopus v4, an evolution of the Octopus v1, v2, and v3 models, excels in selection and parameter understanding and reformatting. Additionally, we explore the use of graph as a versatile data structure that effectively coordinates multiple open-source models by harnessing the capabilities of the Octopus model and functional tokens.


Hello Community,
Would you like to see Illusion Diffusion in Video format. https://huggingface.co/spaces/AP123/IllusionDiffusion

Let me Know.

nijatzeynalov ijohn07 victor Nick088 SvCy NHLOCAL fffiloni Aldyytzyy maxvirusvj


Chunte not-lain dillfrescott Joseph717171


Dear all,

We are happy to share that we have just open-sourced over 200 low-bit LLMs. For the MLX community, we have prepared 2-4 bit versions of mainstream LLMs. You can visit the following collection to access them: https://huggingface.co/collections/GreenBitAI/greenbitai-mlx-llm-6614eb6ceb8da657c2b4ed58.

These low-bit models can be conveniently used through our open-source tool at https://github.com/GreenBitAI/gbx-lm.

Compared to other open-source quantization algorithms, these models provide better accuracy retention. We have provided some model evaluation results here:
https://github.com/GreenBitAI/green-bit-llm/blob/main/green_bit_llm/evaluation/README.md.

You can also evaluate the models yourself using the evaluation script we provided.

YaTharThShaRma999 victor yanghaojin alobnayis NicoNico KingNish adhisetiawan clem


two new VLM benchmarks! 🤩

BLINK: evaluates tasks that humans can solve within a blink 👀 https://huggingface.co/datasets/BLINK-Benchmark/BLINK

SEED-2-Plus: multichoice questions on charts, maps, webs 😍 https://huggingface.co/datasets/AILab-CVC/SEED-Bench-2-plus


text-generation-inference 2.0.2 is out.

- Native support for Idefics2, with much better efficiency than llava 1.6 (next) !

Phi3, Increase VLM support in the openai layer.

Release notes https://github.com/huggingface/text-generation-inference/releases/tag/v2.0.2


EMOPortraits: Emotion-enhanced Multimodal One-shot Head Avatars (2404.19110)

🚀🎭🌟 New Research Alert - CVPR 2024 (Avatars Collection)! 🌟🎭🚀
📄 Title: EMOPortraits: Emotion-enhanced Multimodal One-shot Head Avatars 🔝

📝 Description: EMOPortraits is an enhanced multimodal one-shot head avatar model that achieves SOTA performance in emotion transfer and audio-driven facial animation tasks by improving the training pipeline and architecture to better handle intense and asymmetric facial expressions, while also proposing a novel multiview video dataset containing a wide range of such expressions.

👥 Authors: Nikita Drobyshev, Antoni Bigata Casademunt, Konstantinos Vougioukas, Zoe Landgraf, Stavros Petridis, and Maja Pantic

📅 Conference: CVPR, Jun 17-21, 2024 | Seattle WA, USA 🇺🇸

📄 Paper: https://huggingface.co/papers/2404.19110

🌐 GitHub Page: https://neeek2303.github.io/EMOPortraits

📚 More Papers: more cutting-edge research presented at other conferences in the https://huggingface.co/spaces/DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin

🚀 Added to the Avatars Collection: https://huggingface.co/collections/DmitryRyumin/avatars-65df37cdf81fec13d4dbac36

🔍 Keywords: #EMOPortraits #EmotionalTransfer #FacialAnimation #HeadAvatar #MultimodalLearning #OneShotLearning #AsymmetricFacialExpressions #IntenseFacialExpressions #NovelDataset #CVPR2024 #DeepLearning #ComputerVision #Innovation

DmitryRyumin KingNish victor Delik jeffboudier RedSparkie AlekseiPravdin nroggendorff


Really excited to read about Kolmogorov Arnold Networks as a novel alternatives to Multi Layer Perceptrons.

Excerpt:
> Kolmogorov-Arnold Networks (KANs) are promising alternatives of Multi-Layer Perceptrons (MLPs). KANs have strong mathematical foundations just like MLPs: MLPs are based on the universal approximation theorem, while KANs are based on Kolmogorov-Arnold representation theorem. KANs and MLPs are dual: KANs have activation functions on edges, while MLPs have activation functions on nodes. This simple change makes KANs better (sometimes much better!) than MLPs in terms of both model accuracy and interpretability.

https://github.com/KindXiaoming/pykan


⌚ Visiting the past with Time Machine GPT!

We are all familiar with the concept of a suite of models being a series of variants of a certain model that differ mainly in size. For example, Llama-2 7B, Llama-2 13B, Llama-2 70B

But this is not always the case. Researchers from The University of Oxford, The Alan Turing Institute, and The University of Manchester introduced TimeMachineGPT (TiMaGPT), a suite of language models that were pretrained on data constrained by a certain period in time. Instead of various sizes of the model, you get the same model but trained on different data coming from different times.

Using a GPT-2 model architecture with 117 million parameters, they trained 12 different models on Wikipedia and WMT News from 2011 to 2022 with each year represented by a model. For example, TiMaGPT-2011, TiMaGPT-2012, ..., TiMaGPT-2022.

🤔 But how could these models be useful?

They can be very useful. For example:

1⃣ Most language models are static in the sense that they are trapped in the time bubble of their pretraining data, their knowledge is limited by the cut-off date of their training dataset. In order to update their knowledge, Temporal Adaptation can be performed, which means further training on newer data. The TiMaGPT series of models can be used to study the limitations of Temporal Adaptation of language models.

2⃣ Word meaning can change not only with its context but also with its time of use and there is a large amount of research that focuses on understanding how embeddings shift through time. TiMaGPT will be very helpful in studying this phenomenon.

3⃣ One more use case in the context of Time-series forecasting and event prediction is "backtesting". Which is using historical data to evaluate new models for forecasting the future. Models like TiMaGPT (each living in its own time without any knowledge of the future/present) will be great for such a use case.

🤗 All models and datasets are on the hub: https://huggingface.co/Ti-Ma

samusenps monsoon-nlp maywell JackCloudman adamelliotfields


🚀 just reached 3K+ readers on this blog post about RAG using only HF🤗 related tools in just a little over 1 week from publishing.

📃the most interesting thing about it is that you can use the FAISS index in the datasets library to retrieve your most similar documents.

🔗https://huggingface.co/blog/not-lain/rag-chatbot-using-llama3

Happy reading everyone ✨


𝗔𝗯𝗼𝘂𝘁 𝘁𝗵𝗲 𝗧𝗧𝗦 𝗔𝗿𝗲𝗻𝗮

𝗔𝗯𝗼𝘂𝘁 𝘁𝗵𝗲 𝗧𝗧𝗦 𝗔𝗿𝗲𝗻𝗮

𝗢𝗽𝗲𝗻𝗩𝗼𝗶𝗰𝗲 𝗩𝟮

𝗢𝗽𝗲𝗻𝗩𝗼𝗶𝗰𝗲 𝗩𝟮

𝗖𝗼𝗺𝗽𝗮𝗿𝗲 𝘁𝗵𝗲 𝗺𝗼𝗱𝗲𝗹𝘀 𝗼𝗻 𝘁𝗵𝗲 𝗧𝗧𝗦 𝗔𝗿𝗲𝗻𝗮:

𝗖𝗼𝗺𝗽𝗮𝗿𝗲 𝘁𝗵𝗲 𝗺𝗼𝗱𝗲𝗹𝘀 𝗼𝗻 𝘁𝗵𝗲 𝗧𝗧𝗦 𝗔𝗿𝗲𝗻𝗮:

Excited to launch two new SOTA text-to-speech models on the TTS Arena:

- OpenVoice V2
- Play.HT 2.0

𝗔𝗯𝗼𝘂𝘁 𝘁𝗵𝗲 𝗧𝗧𝗦 𝗔𝗿𝗲𝗻𝗮

The TTS Arena is an open sourced Arena where you can enter a prompt, have two models generate speech, and vote on which one is superior.

We compile the results from the votes into a automatically updated leaderboard to allow developers to select the best model.

We've already included models such as ElevenLabs, XTTS, StyleTTS 2, and MetaVoice. The more votes we collect, the sooner we'll be able to show these new models on the leaderboard and compare them!

𝗢𝗽𝗲𝗻𝗩𝗼𝗶𝗰𝗲 𝗩𝟮

OpenVoice V2 is an open-sourced speech synthesis model created by MyShell AI that supports instant zero-shot voice cloning. It's the next generation of OpenVoice, and is fully open-sourced under the MIT license.
https://github.com/myshell-ai/OpenVoice

𝗣𝗹𝗮𝘆.𝗛𝗧 𝟮.𝟬

Play․HT 2.0 is a high-quality proprietary text-to-speech engine. Accessible through their API, this model supports zero-shot voice cloning.

𝗖𝗼𝗺𝗽𝗮𝗿𝗲 𝘁𝗵𝗲 𝗺𝗼𝗱𝗲𝗹𝘀 𝗼𝗻 𝘁𝗵𝗲 𝗧𝗧𝗦 𝗔𝗿𝗲𝗻𝗮:

https://huggingface.co/spaces/TTS-AGI/TTS-Arena


Replacing Judges with Juries: Evaluating LLM Generations with a Panel of
  Diverse Models (2404.18796)

As part of the Data is Better Together MPEP project, we are now at the point where some translation efforts have successfully translated 500 highly ranked prompts into a new target language (amazing work from @Rijgersberg et al!)

Our next step is to use these translated prompts to evaluate the performance of LLMs for non English languages.

Does LLM, as a judge, work outside of English?

Ideally, it would be compelling to leverage LLMs to judge models for non-English since this significantly lowers the barrier to evaluating models (although it doesn't remove this barrier altogether).

What we want to know is:
- does auto/LLM eval work in general for a particular language
- which model(s) works best as a judge
- do LLMs' judgments of non-English models match human preferences?

We're starting to think about how to approach this. If you have any ideas of possible approaches feel free to comment or join the discussion here: https://github.com/huggingface/data-is-better-together/issues/61

Other ideas...

Could an approach like https://huggingface.co/papers/2404.18796 with the SOA models for a particular language work? i.e., choose 4 of the best open LLMs for Arabic and use those at the pool of raters rather than relying on one powerful judge LLM?

NePe kristaller486 mmhamdy Rijgersberg alvarobartt


It’s exciting to see Apple’s commitment to opensource AI research lately. From a new awesome machine learning framework (mlx) to a family of purely open models (openELM) and incredibly visionary papers (LLMs in a flash, MM1) not mention the vibrant OSS community behind mlx - All alpha signs of something huge dropping in this year’s #AppleEvent & #WWDC

victor onandon adorkin julien-c alielfilali01 luancloud


HyperSD released the 8-steps CFG-preserved LoRA just now, may be the first acceleration plugin that preserves the original CFG for both SDXL and SD15, you can adjust your negative prompts now!!!
Hyper-SDXL-8steps-CFG-LoRA: https://huggingface.co/ByteDance/Hyper-SD/blob/main/Hyper-SDXL-8steps-CFG-lora.safetensors
Hyper-SD15-8steps-CFG-LoRA: https://huggingface.co/ByteDance/Hyper-SD/blob/main/Hyper-SD15-8steps-CFG-lora.safetensors

YaTharThShaRma999 victor radames OzzyGT multimodalart


OpenELM in Core ML

Apple recently released a set of efficient LLMs in sizes varying between 270M and 3B parameters. Their quality, according to benchmarks, is similar to OLMo models of comparable size, but they required half the pre-training tokens because they use layer-wise scaling, where the number of attention heads increases in deeper layers.

I converted these models to Core ML, for use on Apple Silicon, using this script: https://gist.github.com/pcuenca/23cd08443460bc90854e2a6f0f575084. The converted models were uploaded to this community in the Hub for anyone that wants to integrate inside their apps: https://huggingface.co/collections/corenet-community/openelm-core-ml-6630c6b19268a5d878cfd194

The conversion was done with the following parameters:
- Precision: float32.
- Sequence length: fixed to 128.

With swift-transformers (https://github.com/huggingface/swift-transformers), I'm getting about 56 tok/s with the 270M on my M1 Max, and 6.5 with the largest 3B model. These speeds could be improved by converting to `float16`. However, there's some precision loss somewhere and generation doesn't work in `float16` mode yet. I'm looking into this and will keep you posted! Or take a look at this issue if you'd like to help: https://github.com/huggingface/swift-transformers/issues/95

I'm also looking at optimizing inference using an experimental kv cache in swift-transformers. It's a bit tricky because the layers have varying number of attention heads, but I'm curious to see how much this feature can accelerate performance in this model family :)

Regarding the instruct fine-tuned models, I don't know the chat template that was used. The models use the Llama 2 tokenizer, but the Llama 2 chat template, or the default Alignment Handbook one that was used to train, are not recognized. Any ideas on this welcome!

ZachNagengast brightbit reach-vb victor radames JimRWallace NickyNicky 1duo HelloCephalopod not-lain julien-c MathisDevFP maxrubin629 michellenicole Rainierraoul anthonymikinka toddmath irotem98 velyan

reach-vb JimRWallace SuperL3D HelloCephalopod not-lain azhka MathisDevFP toddmath velyan

reach-vb HelloCephalopod not-lain MathisDevFP Norod78 0xjorgev


Model                                             / Avg. (pass@3)

Model                                             / Avg. (pass@3)

Claude 3 Haiku                          / 27.67

Claude 3 Haiku                          / 27.67

PECC: Problem Extraction and Coding Challenges (2404.18766)

How Robust Is Your Model in Complex Code Generation Tasks? 🤔

We've launched the PECC benchmark to challenge chat models in code generation, drawing from the Advent of Code for programming tasks and the Euler Project for math-heavy challenges. This new task tests models with problems presented in both detailed prose and concise "leet code" styles, evaluating their ability to understand and solve complex coding issues and math problem in chat-based interactions.

It seems that the Claude 3 models outperforme ChatGPT:
Model                                             / Avg. (pass@3)
Claude 3 Haiku                          / 27.67
GPT-3.5-Turbo                            / 23.75
Mixtral-8x22B-Instruct-v0.1  / 8.35

Read our Preprint📃: https://huggingface.co/papers/2404.18766
Look at the dataset🔎: https://huggingface.co/datasets/PatrickHaller/pecc

We also got accepted at LREC-COLING '24  🎉


Introducing JARVIS Tony's voice assistant for You.

JARVIS responds to all your questions in audio format.
Must TRY -> https://huggingface.co/spaces/KingNish/JARVIS

Jarvis is currently equipped to accept text input and provide audio output.
In the future, it may also support audio input.

DEMO Video:

victor sebastianking SvCy DamarJati Ezi Marc-Gloaguen julien-c


I've built a custom component that integrates Rerun web viewer with Gradio, making it easier to share your demos as Gradio apps.

Basic snippet
```
# pip install gradio_rerun gradio
import gradio as gr
from gradio_rerun import Rerun

gr.Interface(
    inputs=gr.File(file_count="multiple", type="filepath"),
    outputs=Rerun(height=900),
    fn=lambda file_path: file_path,
).launch()
```
More details here https://huggingface.co/spaces/radames/gradio_rerun
Source https://github.com/radames/gradio-rerun-viewer

Follow Rerun here https://huggingface.co/rerun

louisbrulenaudet AlekseiPravdin andreasnaoum Tyfo87


💰❌ 𝐑𝐞𝐬𝐞𝐚𝐫𝐜𝐡 𝐟𝐨𝐫 𝐭𝐡𝐞 𝐯𝐞𝐫𝐲 𝐆𝐏𝐔 𝐏𝐨𝐨𝐫 - 𝐒𝐜𝐚𝐥𝐢𝐧𝐠 𝐥𝐚𝐰𝐬 𝐫𝐞𝐩𝐥𝐢𝐜𝐚𝐭𝐢𝐨𝐧

🎆 Good news: 𝘆𝗼𝘂 𝗰𝗮𝗻 𝗱𝗼 𝗰𝘂𝘁𝘁𝗶𝗻𝗴-𝗲𝗱𝗴𝗲 𝗿𝗲𝘀𝗲𝗮𝗿𝗰𝗵 𝘄𝗶𝘁𝗵 𝗮 𝗰𝗮𝗹𝗰𝘂𝗹𝗮𝘁𝗼𝗿 𝗮𝗻𝗱 𝗠𝗶𝗰𝗿𝗼𝘀𝗼𝗳𝘁 𝗣𝗮𝗶𝗻𝘁 𝟮𝟬𝟬𝟲!

The Chinchilla experiments (by Google DeepMind) ran hundreds of pre-trainings with models >1B parameters (I do not want to imagine how much that cost) to 𝗳𝗶𝗻𝗱 𝘁𝗵𝗲 𝗼𝗽𝘁𝗶𝗺𝗮𝗹 𝗿𝗮𝘁𝗶𝗼 𝗼𝗳 𝗺𝗼𝗱𝗲𝗹 𝘀𝗶𝘇𝗲 𝘃𝘀 𝘁𝗿𝗮𝗶𝗻𝗶𝗻𝗴 𝘁𝗼𝗸𝗲𝗻𝘀. Why is this question so important?
Well, you only ever have access to a fixed compute, counted in FLOPs (floating point operations). So if your model is bigger, you will have less compute to train on many tokens, and if you want to train on more tokens, your model will be smaller. When model trainings cost million, you absolutely need to get this right.

The new paper "Chinchilla Scaling: A replication attempt" by Epoch AI sets on on the ambitious goal of reproducing this.

But since the authors do not have infinite money, they decided to directly run their computations from DeepMind's own experiments! They took the figure from the last experiment (cf slide below), measured point positions, picked color codes, and ended up reconstructing the underlying data.

💥 They then just fit the scaling laws proposed by the Chinchilla Authors, but arrived at wildly different results! They find that as a rough rule of thumb, you should use 20 training tokens for each parameter in your model, instead of the 70 obtained in the original paper. They also point out inconsistencies in the paper, and unrealistically narrow confidence intervals.

➡ This only contradicts the results from the last (out of 3) experiments in the Chinchilla paper. And the model trained at the end of the Chinchilla paper still seems properly scaled.

✅ But it does show that a tiny bit more theoretical work can go a long way, especially given the huge financial costs that such an error can have!

chansung hllj maywell radames victor cstr

maywell radames louisbrulenaudet KingNish NotSiDDH


Adding a long prompt can help you fight LLM hallucinations. However, if you know exactly how you want your LLM output constrained, there are much better strategies! 💪

Did you know you can force your LLM to ALWAYS generate a valid JSON file? Or to follow a well-defined answer template? You can do that and more with the 🤗 transformers-compatible `outlines` library.

It doesn't only allow you to master your LLM -- your text generation application will also become faster! 🔥 The more constrained your text generation is, the bigger speedups you'll see!

Follow @remi and other `outlines` folks to stay on top of the constrained generation game 🧠


The hype is real: a mysterious gpt2-chatbot model has appeared on the LLM Arena Leaderboard 👀.
It seems to be at least on par with the top performing models (closed and open).

To try it out: https://chat.lmsys.org/ -> then click on the Direct Chat tab and select gpt2-chatbot.

 Take your bet, what do you think it is?

KingNish FreeHugsForRobots radames mvaloatto AlekseiPravdin alfredplpl Lin-Rexter icpro adumred not-lain

KingNish radames xiaotianhan louisbrulenaudet AlekseiPravdin PatrickHaller not-lain


Should media organizations strike deals with big tech companies? Here are two colliding news stories about licensing:

1. The Financial Times has secured a licensing agreement with OpenAI to license its material both for training and queries on ChatGPT. It is the fifth such deal, following similar agreements with Associated Press, Axel Springer, Le Monde and Prisa Media. "Financial terms were not disclosed."

"Apart from the benefits to the FT, there are broader implications for the industry. It’s right, of course, that AI platforms pay publishers for the use of their material. OpenAI understands the importance of transparency, attribution, and compensation – all essential for us."

2. Meanwhile, French media outlet Mediapart is refusing to cash in money from Google, which it is entitled to under so-called "neighbouring rights" for the right to display their news content online.

Why? Due to issues with disclosing financial terms: "The confidentiality clauses imposed by Google today prevent us from publicizing to our readers not only the total amount paid, but also the amount Mediapart is entitled to receive."

"In our view, financial dependence on platforms is incompatible with our public service mission, which is to make the powerful face up to their responsibilities. It also seems extremely dangerous economically."

Two positions at opposite sides of the spectrum.

- The Financial Times and OpenAI strike content licensing deal
https://www.ft.com/content/33328743-ba3b-470f-a2e3-f41c3a366613

- Droits voisins : Mediapart lance la bataille de la transparence contre Google (in French) https


Am I the only one who think command-r-+ is a better daily Assistant than ChatGPT-4? (and it's not even close :D)

KingNish jgitsolutions fdaudens radames not-lain WaveCut Alagib Rybens DamarJati Sachiii84


#ICLR 2024 is almost there 🔥🔥🔥 counting the days to be again in the beautiful city of Vienna participating in the The Twelfth International Conference on Learning Representations, hope to see many of the Hugging Face comunity there!

I would like to contribute 🎁 by releasing the second Knowledge Vault, with 100 lectures visualized from the last 10 years of ICLR from 2014 to 2023, including knowledge graphs for all the Invited Lectures and some extras, with almost 3000 topics represented. (Of course using several AI tools including Llama3)

You can explore it here:
🌏 https://theendofknowledge.com/Vaults/2/ICLR2014-2023.html

And you can learn more about the Vaults here:
📝https://www.linkedin.com/pulse/knowledge-vaults-david-vivancos-lbjef/

Hope you like the Knowledge Vault!

victor notoookay AlekseiPravdin damerajee


If you are into video generation you should check this great Space by @KingNish - generate videos in ~10 seconds!

https://huggingface.co/spaces/KingNish/Instant-Video


https://arxiv.org/pdf/2402.06852
Technical report for ChemLLM 1.5 released.🤗


🦢 The SWIM-IR dataset contains 29 million text-retrieval training pairs across 27 diverse languages. It is one of the largest synthetic multilingual datasets generated using PaLM 2 on Wikipedia! 🔥🔥

SWIM-IR dataset contains three subsets :
- Cross-lingual:`nthakur/swim-ir-cross-lingual`
- Monolingual: `nthakur/swim-ir-monolingual`
- Indic Cross-lingual: `nthakur/indic-swim-ir-cross-lingual`

Check it out:
https://huggingface.co/collections/nthakur/swim-ir-dataset-662ddaecfc20896bf14dd9b7


🔥 Transfer model's Chat feature, Context length and Knowledge to another under 1 minute without any train.

Imagine being able to create chat models, expand context, and transfer domain-specific knowledge to models, all within a matter of minutes. Our innovative approach, based on a combination of diff-based techniques and sigmoid ratio calculations, makes this possible.

By considering the diffs between the desired information model (long context or chat) and the base model, as well as the diffs between the base model and the target model, we can efficiently transfer features and expand context without the need for extensive training or resources.

Our method minimizes model degradation and ensures that only the desired information is captured, resulting in high-quality models that can be created with just a single click. Whether you need a chat model, expanded context, or domain-specific knowledge transfer, our approach offers a rapid and effective solution.

In blog post below, we will dive into the details of our method, provide code examples, and showcase the impressive results achieved using our approach. Get ready to revolutionize your model creation process and unlock new possibilities with this powerful technique.

Blog - https://huggingface.co/blog/maywell/llm-feature-transfer

maywell IHaBiS lunarflu Unggi beomi victor julien-c FengJ DeathGodlike hivaze kaki-paper hiauiarau

IHaBiS lunarflu beomi celarlin scoop777 sappho192 kaki-paper


Soon new releases on NeverSleep 👀
8B/70B Llama3 RP fine-tune in the work!

IkariDev Ainonake saishf xxx777xxxASD bluuwhale Andrewwwwww joujiboi lunarflu Nacholmo antiven0m EveryPizza Blobbinski ZootZootTesla beberik victor WesPro kleberbaum den0620 TeoR4eg Tillx85 HailJebus ABX-AI

ggnick Ainonake Andrewwwwww Th3Nomad lunarflu s0lu Meggido ZootZootTesla bharathchalla MrHillsss den0620 Kenshiro-28 ABX-AI


I created a Twitter account a while back. I finally decided to make it public SebastianG74019. For those of you following @Locutusque on Twitter, that is not me! 😂


VTuber Logo Generator❤🪄⭐ by @gojiteji
https://huggingface.co/spaces/gojiteji/VTuberLogoGenerator
How this works:
- mistralai/Mixtral-8x7B-Instruct-v0.1 for Japanese transliteration.
- Stable Diffusion 3 for logo generation.
- simple k-means for color selection.

KvrParaskevi dillfrescott lunarflu grx96 victor julien-c willowill5 pabloce not-lain

coderjuzi dillfrescott lunarflu not-lain


New update to mlx-rag-gguf:
- mlx supported phi-3-mini-4k gguf  weight.
- support for other gguf weights (llama arch) 4 & 8 bits quantized.
repo: https://github.com/Jaykef/mlx-rag-gguf
model https://huggingface.co/Jaward/phi-3-mini-4k-instruct.Q4_0.gguf


zaursamedov1 dgallitelli ameerazam08 gojiteji svjack lunarflu victor Yasmanyel3men2 not-lain danielus


LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding (2404.16710)

Layer Skip

Enabling Early Exit Inference and Self-Speculative Decoding

https://huggingface.co/papers/2404.16710

We present LayerSkip, an end-to-end solution to speed-up inference of large language models (LLMs). First, during training we apply layer dropout, with low dropout rates for earlier layers and higher dropout rates for later layers, and an early exit loss where all transformer layers share the same exit. Second, during inference, we show that this training recipe increases the accuracy of early exit at earlier layers, without adding any auxiliary layers or modules to the model. Third, we present a novel self-speculative decoding solution where we exit at early layers and verify and correct with remaining layers of the model. Our proposed self-speculative decoding approach has less memory footprint than other speculative decoding approaches and benefits from shared compute and activations of the draft and verification stages. We run experiments on different Llama model sizes on different types of training: pretraining from scratch, continual pretraining, finetuning on specific data domain, and finetuning on specific task. We implement our inference solution and show speedups of up to 2.16x on summarization for CNN/DM documents, 1.82x on coding, and 2.0x on TOPv2 semantic parsing task.

Squadgroup1234 Chunte gojiteji samadpls yizhilll travisking QuocKhanh vincentmichael089


Wow i can post on HF now!
Love HF so much 🤗❤

Chunte gojiteji Moibe julien-c not-lain Tonic SvCy DavidGF


Explore the Latest Top Papers with Papers Leaderboard!
We are excited to introduce a new way to explore the most impactful research papers: Papers Leaderboard! This feature allows you to easily find the most talked-about papers across a variety of fields.
Hf-demo : https://huggingface.co/spaces/ameerazam08/Paper-LeaderBoard
Happy weekends!


Yay we got 500K+ monthly HF downloads on our Unsloth HF repo! :) Super appreciate everyone in the OSS community - and thanks for using Unsloth!!

clem giux78 lunarflu nbroad alielfilali01 Kukedlc Oscar007 Dlbk Chunte catastropiyush AtAndDev julien-c duxx damerajee feeltheAGI beomi

clem lunarflu fredericmenezes samusenps nbroad alielfilali01 Kukedlc Chunte AtAndDev f0ster julien-c louisbrulenaudet damerajee aloobun beomi

Kukedlc Oscar007 AtAndDev damerajee Ji-Xiang beomi


clem lunarflu samusenps Dlbk julien-c not-lain zaursamedov1 GPT007


Glad to see Idefics2 making its way into the awesome OpenVLM Leaderboard which ranks VLMs. 🏆
2nd in its category (<10B parameters and open weights)!

While InternLM-XComposer2 uses proprietary data, Idefics2 is built solely using openly available data.

Leaderboard: https://huggingface.co/spaces/opencompass/open_vlm_leaderboard
Model: https://huggingface.co/HuggingFaceM4/idefics2-8b

julien-c yjernite lunarflu not-lain Cuiunbo Dlbk fdaudens lewtun Narsil London12345 kaifahmad orrzohar


Benchmarks!

I have lately been diving deep into the main benchmarks we all use to evaluate and compare models.

If you've never actually looked under the hood for how benchmarks work, check out the LM eval harness from EleutherAI: https://github.com/EleutherAI/lm-evaluation-harness

+ check out the benchmark datasets, you can find the ones for the LLM leaderboard on the about tab here: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard, then click the dataset and actually peak at the data that comprises these benchmarks.

It feels to me like benchmarks only represent a tiny portion of what we actually use and want LLMs for, and I doubt I'm alone in that sentiment.

Beyond this, the actual evaluations of responses from models are extremely strict and often use even rudimentary NLP techniques when, at this point, we have LLMs themselves that are more than capable at evaluating and scoring responses.

It feels like we've made great strides in the quality of LLMs themselves, but almost no change in the quality of how we benchmark.

If you have any ideas for how benchmarks could be a better assessment of an LLM, or know of good research papers that tackle this challenge, please share!

julien-c Dlbk lewtun dillfrescott mmhamdy Noomam welkson


So hard to keep up with pace!!! Lots of new Chinese fine-tunes are being released on  HF

So I asked my agent to create a collection
https://huggingface.co/collections/xianbao/llama3-zh-662ba8503bdfe51948a28403

code: https://colab.research.google.com/drive/1ap6fP-VytZE367Nqk26DeQqgQkYaf-cD#scrollTo=eljRbYb4c92M

Would be nice to run then regularly. Any thoughts / suggestions on where to host this cron job?

victor jharshraj Jaye13 Dlbk fdaudens svjack Tonic


How do Microsoft and Alphabet (Google) results compare?

Microsoft Reports Rising Revenues as A.I. Investments Bear Fruit
- 17 % jump in revenue and a 20 % increase in profit for the first three months of the year.
- Revenue was $61.9 billion, up from $52.9 billion a year earlier.
- Profit hit $21.9 billion, up from $18.3 billion.
- More than a fifth of that growth came from its generative A.I. services
https://www.nytimes.com/2024/04/25/technology/microsoft-earnings.html

Alphabet’s Revenue Jumps 15% to $80.5 Billion
- $80.5 billion in quarterly sales, up 15 % from a year earlier. Profit climbed 36 % to $23.7 billion.
- For the first time, a dividend of 20 cents per share
- It spent $12 billion on capital expenditures in the first quarter, soaring 91 % from a year earlier.
https://www.nytimes.com/2024/04/25/technology/alphabet-earnings.html

Meta’s Open Source Llama 3 Is Already Nipping at OpenAI’s Heels - Wired
- "if open source models prove competitive, developers and entrepreneurs may decide to stop paying to access the latest model from OpenAI or Google and use Llama 3 or one of the other increasingly powerful open source models that are popping up."
- "Open models appear to be dropping at an impressive clip."
https://www.wired.com/story/metas-open-source-llama-3-nipping-at-openais-heels/


Announcing that we are on our way to solve a long standing issue of document processing: correction of OCR mistakes. Pleias publishes the largest dataset to date with automated OCR correction, 1 billion words in English, French, German and Italian.

OCR quality is long-standing issue of digitization. Cultural heritage texts are especially concerned due to the primary sources being old documents (with many artifacts, blots, degradation) and to the limitation of OCR technology for historical scripts. When we released Common Corpus, a 500 Billion words corpus in the public domain, this was the primary criticism.

Recent breakthrough in post-OCR correction has been made possible thanks to progress in open LLM research and several months of dedicated training and alignment by Pleias as well as the HPC resources from GENCI–IDRIS (Grant 2023-AD011014736) on Jean-Zay.

Announcement: https://huggingface.co/blog/Pclanglais/post-ocr-correction

Post-OCR-Correction dataset: https://huggingface.co/datasets/PleIAs/Post-OCR-Correction

fdaudens davanstrien andrewrreed jharshraj Dlbk AkimfromParis louisbrulenaudet mexicanamerican VictorSanh victor Molbap clem J-Hansen LucasThil


Татар теле көне белән!

Татар теле көне белән!

Today, April 26, is the Day of the Tatar Language! 🌟
To celebrate, we release our new language model, Tweety Tatar 🐣

https://huggingface.co/Tweeties/tweety-tatar-base-7b-2024-v1

The model was converted from Mistral Instruct v0.2 using a novel technique called trans-tokenization. As a result, the model uses a brand-new tokenizer, fully tailored for the Tatar language.

We also release a model which can be finetuned for translation of English or Russian into Tatar, and achieves a performance similar to commercial offerings:

https://huggingface.co/Tweeties/tweety-tatar-hydra-base-7b-2024-v1

More details in our upcoming paper 👀
François REMY, Pieter Delobelle, Alfiya Khabibullina

Татар теле көне белән!

justalphie pdelobelle monsoon-nlp murat Moibe IPSAN midwestcyr


Finally, Qwen1.5-110B is out! With weights and demo!

Blog: https://qwenlm.github.io/blog/qwen1.5-110b/
Demo: https://huggingface.co/spaces/Qwen/Qwen1.5-110B-Chat-demo
Base: https://huggingface.co/Qwen/Qwen1.5-110B
Chat: https://huggingface.co/Qwen/Qwen1.5-110B-Chat

This model has some specific features:
* GQA
* 32K token context length
* Multilingual support

We feel good about its performance on benchmarks, including those for base models and chat models, but we still need more of your testing and feedback to help us know its capabilities and limitations!

Additionally, the base model has not learned chatml tokens. Yeah if you use chatml format, you need to be careful about it!

Enjoy and stay tuned for Qwen2!

louisbrulenaudet AdinaY hiyouga tomaarsen xianbao lunarflu martineden Chunte lewtun svjack

AdinaY hiyouga tomaarsen fdaudens xianbao lunarflu Chunte lewtun


ConsistentID: Portrait Generation with Multimodal Fine-Grained Identity
  Preserving (2404.16771)

🚀🎭🔥 New Research Alert (Avatars Collection)! 🔥🎭🚀
📄 Title: ConsistentID: Portrait Generation with Multimodal Fine-Grained Identity Preserving 🔝

📝 Description: ConsistentID is a novel portrait generation method that preserves the fine-grained identity of a single reference image.

👥 Authors: Jiehui Huang et al.

🔗 Paper: https://huggingface.co/papers/2404.16771

🌐 Github Page: https://ssugarwh.github.io/consistentid.github.io/
📁 Repository: https://github.com/JackAILab/ConsistentID

📚 More Papers: more cutting-edge research presented at other conferences in the https://huggingface.co/spaces/DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin

🚀 Added to the Avatars Collection: https://huggingface.co/collections/DmitryRyumin/avatars-65df37cdf81fec13d4dbac36

🔍 Keywords: #ConsistentID #PortraitGeneration #IdentityPreservation #FineGrainedControl #ImageSynthesis #GenerativeModels  #ComputerVision #DeepLearning


🦙🦙 LLaMA Duo project update

Last time, I gave a brief introduction about LLaMA Duo project with @sayakpaul . It is a simple toolset to aligning sLLM with service LLM with coverage dataset 👉🏻 (https://huggingface.co/posts/chansung/708646454991943).
- coverage dataset is what we believe to be the most important/desired (instruction, response) pairs. In system thinking, each instruction could be an analogy of a function from traditional programming. We make unit tests and measure the coverage % for all the features/functions. Similarly, we need to ensure if our fine-tuned model could handle what % of given instructions from coverage dataset satisfactory (hence coverage dataset).

We have tested it with "Coding" category of data from https://huggingface.co/datasets/HuggingFaceH4/no_robots dataset. It has about 300 SFT training data points under Coding category. After fine-tuning Gemma 7B model on that, the result was very poor. LLaMA Duo's evaluation tool gave < 20% of metrics in similarity and preciseness on the test split.

So, we used LLaMA Duo's synthetic data generation tool to generate 60k data points that looks similar to the original dataset. We first created ~10k synthetic data points, then created 50k more based on the synthetic dataset itself.

After fine-tuning Gemma 7B on the 60k synthetic dataset, the evaluation result went up to 80~90% high. Also, when testing out the model in UI, it tends to give good responses.

It is a good showcase to transition from service LLM to sLLM or having a backup sLLM for service LLM failure scenarios. I am going to expand this experiments on all categories of no_robots dataset. It will roughly generate > 100k data points.

Here are some links:
- LLaMA Duo project repo: https://github.com/deep-diver/llamaduo
- 60k Coding synthetic dataset: https://huggingface.co/datasets/chansung/merged_ds_coding
- Fine-tuned Gemma 7B model: https://huggingface.co/chansung/coding_llamaduo_60k_v0.2

Tonic peterschmidt85 chansung lunarflu juyongjiang

hllj chansung lunarflu Hiraishin juyongjiang


CatLIP: CLIP-level Visual Recognition Accuracy with 2.7x Faster
  Pre-training on Web-scale Image-Text Data (2404.15653)

CatLIP

CLIP-level Visual Recognition Accuracy with 2.7x Faster Pre-training on Web-scale Image-Text Data

https://huggingface.co/papers/2404.15653

Contrastive learning has emerged as a transformative method for learning effective visual representations through the alignment of image and text embeddings. However, pairwise similarity computation in contrastive loss between image and text pairs poses computational challenges. This paper presents a novel weakly supervised pre-training of vision models on web-scale image-text data. The proposed method reframes pre-training on image-text data as a classification task. Consequently, it eliminates the need for pairwise similarity computations in contrastive loss, achieving a remarkable 2.7times acceleration in training speed compared to contrastive learning on web-scale data. Through extensive experiments spanning diverse vision tasks, including detection and segmentation, we demonstrate that the proposed method maintains high representation quality.


Updated the https://huggingface.co/datasets/vikhyatk/lnqa dataset to include images, so you no longer need to separately download them from OpenImages!

beomi clem akashicmarga lunarflu catastropiyush


5 interesting news stories today:

An AI startup made a hyperrealistic deepfake of me that’s so good it’s scary
- "'I think we might just have to say goodbye to finding out about the truth in a quick way,” says Sandra Wachter, a professor at the Oxford Internet Institute"
- "Synthesia uses both large language models and diffusion models to do this. Sees itself as a platform for businesses. Its bet is this: As people spend more time watching videos on YouTube and TikTok, there will be more demand for video content."
- "Synthesia’s policy is to not create avatars of people without their explicit consent. But it hasn’t been immune from abuse."
https://www.technologyreview.com/2024/04/25/1091772/new-generative-ai-avatar-deepfake-synthesia/

WIRED found thousands of ads running on Meta's social platforms promoting sexually explicit "Al girlfriend" apps.
- "Some human sex workers say the platform unfairly polices their own posts more harshly."
- "Many of the virtual women seen in ads reviewed by WIRED are lifelike—if somewhat uncanny—young, and stereotypically pornographic."
https://www.wired.com/story/ads-for-explicit-ai-girlfriends-swarming-facebook-and-instagram/

Wall Street’s Patience for a Costly A.I. Arms Race Is Waning
- "A sell-off in Meta’s stock after the company disclosed huge investments in the technology may be a sign of investor fears about tech giants’ spending."
- "The company plans to spend $35 billion to $40 billion this year — much of that on the technology."
https://www.nytimes.com/2024/04/25/business/dealbook/meta-artificial-intelligence-spending.html

Saudia Arabia Spends Big to Become an A.I. Superpower
https://www.nytimes.com/2024/04/25/technology/to-the-future-saudi-arabia-spends-big-to-become-an-ai-superpower.html

UK competition watchdog steps up scrutiny of big tech’s role in AI startups
https://www.theguardian.com/technology/2024/apr/24/uk-competition-watchdog-steps-up-scrutiny-of-big-techs-role-in-ai-startups-cma-microsoft-amazon

Yasir22 evijit clem noobmldude JvThunder


What needs to go right for an induction head? A mechanistic study of
  in-context learning circuits and their formation (2404.07129)

🔍 Today's pick in Interpretability & Analysis of LMs: by @aadityasingh T. Moskovitz, F. Hill, S. C. Y. Chan, A. M. Saxe (@gatsbyunit)

This work proposes a new methodology inspired by optogenetics (dubbed "clamping") to perform targeted ablations during training to estimate the causal effect of specific interventions on mechanism formation.

Authors use this approach to study the formation of induction heads training a 2L attention-only transformer to label examples via context information.

Notable findings:

- The effects of induction heads are additive and redundant, with weaker heads compensating well for the ablation of a strong induction head in case the latter is ablated.
- Competition between induction heads might emerge as a product of optimization pressure to converge faster, but it is not strictly necessary as all heads eventually learn to solve the task.
- Previous token heads (PTH) influence induction heads in a many-to-many fashion, with any PTH eliciting above-chance prediction from a subsequent induction head
- Three subcircuits for induction are identified, respectively mixing token-label information (1 + 2), matching the previous occurrence of the current class in the context (3qk + 4), and copying the label of the matched class (3v + 5).
- The formation of induction heads is slowed down by a larger number of classes & labels, with more classes and more labels slowing down the formation of the matching and copying mechanisms, respectively. This may have implications when selecting a vocabulary size for LLMs: larger vocabularies lead to an increased compression ratio and longer contexts, but they might make copying more challenging by delaying the formation of induction heads.

💻 Code: https://github.com/aadityasingh/icl-dynamics

📄 Paper: https://huggingface.co/papers/2404.07129

🔍 All daily picks: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-ofc-lms-65ae3339949c5675d25de2f9


Today’s devin’s most difficult task:
build a port of our AutoAgents framework in mlx and develop a demo using a gguf weight - it got close to nailing it (with guidance).

It was magical to witness. I had to take the wheel and help fix some subtle bugs. That said there was still the need for a human software engineer to keep it aligned with the overall goal. Most of my work involved reviewing code, checking shells and alignment chats.

full demo coming soon.

AutoAgents: https://huggingface.co/spaces/LinkSoul/AutoAgents


🤗 Can We Train Chat Models with Raw Data? #1

The idea of training a chat model with desired raw data is incredibly appealing.

However, there is a significant problem with this process. Directly training a chat model with raw data can disrupt its output format.

To solve this issue, the common approach is to create Q/A-formatted datasets. However, this method is time-consuming, costly, and may result in information loss or bias during dataset creation.

So, how can we effectively train raw data? We can utilize the sequential structure of transformer models like Llama, which consists of multiple layers.

I intentionally form the layers responsible for handling the output format in the latter part of the model, and designate the middle to late layers as the starting point for raw training.

You may think that the method involves feeding chat data to the later layers and then training the middle to late layers with raw data, but that's not the case. Such an approach cannot properly address the problem and may even lead to increased model complexity.

The idea presented above doesn't seem bad, so how can we make good use of it? Let's try using a base model.

Read more  - https://huggingface.co/blog/maywell/layer-aware-1

victor sigridjineth obamadronestrikeinu asteriskd


WANDR: Intention-guided Human Motion Generation (2404.15383)

🚀🕺🌟 New Research Alert - CVPR 2024 (Avatars Collection)! 🌟💃🚀
📄 Title: WANDR: Intention-guided Human Motion Generation 🔝

📝 Description: WANDR is a conditional Variational AutoEncoder (c-VAE) that generates realistic motion of human avatars that navigate towards an arbitrary goal location and reach for it.

👥 Authors: Markos Diomataris, Nikos Athanasiou, Omid Taheri, Xi Wang, Otmar Hilliges, Michael J. Black

📅 Conference: CVPR, Jun 17-21, 2024 | Seattle WA, USA 🇺🇸

📄 Paper: https://huggingface.co/papers/2404.15383

🌐 Web Page: https://wandr.is.tue.mpg.de
📁 Repository: https://github.com/markos-diomataris/wandr

📺 Video: https://www.youtube.com/watch?v=9szizM-XUCg

📚 More Papers: more cutting-edge research presented at other conferences in the https://huggingface.co/spaces/DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin

🚀 Added to the Avatars Collection: https://huggingface.co/collections/DmitryRyumin/avatars-65df37cdf81fec13d4dbac36

🔍 Keywords: #WANDR #HumanMotionGeneration #MotionSynthesis #3DAvatar #GoalOrientedMovement #IntentionGuided #ConditionalVAE #CVPR2024 #DeepLearning #Innovation

DmitryRyumin loong0306 Dlbk victor louisbrulenaudet AlekseiPravdin letheviet alicegaz


Open VLM Leaderboard just updated the performance of GPT-4v (20240409), the new proprietary model ranked 1st across 50+ VLMs.  Compared to the pervious version (20231106), the improvements on multimodal perception and reasoning are both huge.

Check the results:
https://huggingface.co/spaces/opencompass/open_vlm_leaderboard


Is it just me or is it real that whenever APPLE releases an open model, they accompany it with a library !? First was MLX, about a month ago AXLEARN and now CORENET ! Could it be just coincidences or does Apple playing some game ? if yes then what is it ... ? What do you think ? maybe i'm just hallucinating now 😅


It's been only a week since I joined 🤗 and the community has released a constant flow of content!

Notable models:
- Apple OpenELM https://huggingface.co/collections/apple/openelm-instruct-models-6619ad295d7ae9f868b759ca + https://huggingface.co/collections/apple/openelm-pretrained-models-6619ac6ca12a10bd0d0df89e
- HuggingFaceM4 Idefics2 https://huggingface.co/HuggingFaceM4/idefics2-8b
- Meta Llama 3 https://huggingface.co/collections/meta-llama/meta-llama-3-66214712577ca38149ebb2b6
- Microsoft Phi-3 https://huggingface.co/collections/microsoft/phi-3-6626e15e9585a200d2d761e3
- Snowflake Arctic https://huggingface.co/collections/Snowflake/arctic-66290090abe542894a5ac520

Great datasets:
- HuggingFaceFW FineWeb https://huggingface.co/datasets/HuggingFaceFW/fineweb
- HuggingFaceM4/the_cauldron https://huggingface.co/datasets/HuggingFaceM4/the_cauldron
- PleIAs/YouTube-Commons https://huggingface.co/datasets/PleIAs/YouTube-Commons

Fascinating Spaces
- InstantMesh https://huggingface.co/spaces/TencentARC/InstantMesh
- Chat with Llama 3 8B https://huggingface.co/spaces/ysharma/Chat_with_Meta_llama3_8b
- Parler-TTS https://huggingface.co/spaces/parler-tts/parler_tts_mini
- AI Jukebox https://huggingface.co/spaces/enzostvs/ai-jukebox
- CosXL https://huggingface.co/spaces/multimodalart/cosxl
- Singing songstarter https://huggingface.co/spaces/nateraw/singing-songstarter
- Play with Idefics2 8B https://huggingface.co/spaces/HuggingFaceM4/idefics-8b
- CodeQwen1.5-7B-Chat Bot👾
https://huggingface.co/spaces/Qwen/CodeQwen1.5-7b-Chat-demo

I expected to be at the center of AI development. I'm not disappointed!

BrigitteTousi clem zealota radames daniel4999 tomaarsen Dlbk andrewrreed vikas danielus Flashyflash evijit lunarflu letheviet 4eyez VictorSanh

BrigitteTousi clem zealota radames tomaarsen lunarflu letheviet 4eyez

BrigitteTousi clem zealota lunarflu letheviet 4eyez

BrigitteTousi clem zealota Midgardsormr lunarflu 4eyez


How to Finetune phi-3 on MacBook Pro
https://huggingface.co/blog/abhishek/phi3-finetune-macbook

abhishek fdaudens BrigitteTousi clem dotdotgod lusstta AlekseiPravdin semigoso


just landed at Hugging Face Hub: community-led computer vision course 📖🤍
learn from fundamentals to details of the bleeding edge vision transformers!

not-lain alvanlii charchits7 mattmdjaga adorkin Lukewood fdaudens BrigitteTousi aidystark clem victor Someman DmitryRyumin sergiopaniego julien-c lunarflu

Tonic charchits7 BrigitteTousi clem victor ak0601 DmitryRyumin Ushio louisbrulenaudet noobmldude lunarflu omaryshchenko rnella01


Live recordings of [AnimateLCM](https://huggingface.co/spaces/wangfuyun/AnimateLCM).


Happy to announce the open source framework to turbo charge devops called patchwork - https://github.com/patched-codes/patchwork

You can use it to build patchflows - workflows that use LLMs for software development tasks like bug fixing, pull request review, library migration and documentation.

Supports any LLM of your choice including our own MoE model - https://huggingface.co/patched-codes/patched-mix-4x7B

Give it a try!

codelion milkowski victor AlekseiPravdin


Got access to Devin today and boy it’s been rocking it - 10x engineer on pure software dev tasks, albeit falls at the mercy of ML/AI tasks. Still a promising work of daring-engineering feat, wishing all the best to the team @cognition_labs

jacob-valdez Abubakr18 tomaarsen victor mariachus Joseph717171


OpenELM: An Efficient Language Model Family with Open-source Training
  and Inference Framework (2404.14619)

OpenELM

An Efficient Language Model Family with Open-source Training and Inference Framework

https://huggingface.co/papers/2404.14619

The reproducibility and transparency of large language models are crucial for advancing open research, ensuring the trustworthiness of results, and enabling investigations into data and model biases, as well as potential risks. To this end, we release OpenELM, a state-of-the-art open language model. OpenELM uses a layer-wise scaling strategy to efficiently allocate parameters within each layer of the transformer model, leading to enhanced accuracy. For example, with a parameter budget of approximately one billion parameters, OpenELM exhibits a 2.36% improvement in accuracy compared to OLMo while requiring 2times fewer pre-training tokens. Diverging from prior practices that only provide model weights and inference code, and pre-train on private datasets, our release includes the complete framework for training and evaluation of the language model on publicly available datasets, including training logs, multiple checkpoints, and pre-training configurations. We also release code to convert models to MLX library for inference and fine-tuning on Apple devices. This comprehensive release aims to empower and strengthen the open research community, paving the way for future open research endeavors.


I have built a Space to compare different vision language model outputs, which model should I add next? 👀
Try them yourself here https://huggingface.co/spaces/merve/compare_VLMs


Fine tune Phi-3 using samatha themed dataset and Huggingface SFT trainer!

In this colab, we simply apply a supervised finetune to phi-3 using the sharegpt format.

```
def formatting_prompts_func(examples):
    convos = examples["conversations"]
    texts = []
    mapper = {"system": "system\n", "human": "\nuser\n", "gpt": "\nassistant\n"}
    end_mapper = {"system": "", "human": "", "gpt": ""}
    for convo in convos:
        text = "".join(f"{mapper[(turn := x['from'])]} {x['value']}\n{end_mapper[turn]}" for x in convo)
        texts.append(f"{text}{EOS_TOKEN}")
    return {"text": texts}

dataset = dataset.map(formatting_prompts_func, batched=True)
print(dataset['text'][8])

```
Opus Samantha consists of 1848 samples with the samantha personality. The dataset covers a wide variety of topics such as logical reasoning, mathematics, legal, and rp.

This notebook serves as a viable option to finetune Phi-3 until Unsloth supports phi-3, which should be very soon. When that happens check out AutoSloth for both SFT, DPO, and langfuse format RAG fine tuning on free tier colab hardware.

Resources:
Dataset: https://huggingface.co/datasets/macadeliccc/opus_samantha
Colab: https://colab.research.google.com/drive/1e8LILflDQ2Me52hwS7uIfuJ9DxE2oQzM?usp=sharing
AutoSloth: https://colab.research.google.com/drive/1Zo0sVEb2lqdsUm9dy2PTzGySxdF9CNkc#scrollTo=bpimlPXVz-CZ

victor KingNish Qwoook midesk pabloce lunarflu andrewatef EddyGiusepe

santyzenith clavel lunarflu Norod78 trtm


Complete Guide to SUPIR Enhancing and Upscaling Images Like in Sci-Fi Movies on Your PC : https://youtu.be/OYxVEvDf284

In this video, I explain how to 1 click install and use the most advanced image upscaler / enhancer in the world that is both commercially and open source available. The upscaler that I am going to introduce you is open source #SUPIR and the model is free to use. SUPIR upscaler is many times better than both paid Topaz AI and Magnific AI and you can use this upscaler on your computer for free forever. The difference of SUPIR vs #Topaz and #Magnific is like ages. So in this tutorial you are going to learn everything about how to install, update and use SUPIR upscaler on your personal computer. The video shows Windows but it works perfectly fine on Linux as well.

Scripts Download Link ⤵
https://www.patreon.com/posts/99176057

Samplers and Text CFG  (Text Guidance Scale) Comparison Link ⤵
https://imgsli.com/MjU2ODQz/2/1

How to install accurate Python, Git and FFmpeg on Windows Tutorial ⤵
https://youtu.be/-NjNy7afOQ0

Full DreamBooth / Fine-tuning Tutorial ⤵
https://youtu.be/0t5l6CP9eBg

Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild : https://arxiv.org/abs/2401.13627

Authors introduce SUPIR (Scaling-UP Image Restoration), a groundbreaking image restoration method that harnesses generative prior and the power of model scaling up. Leveraging multi-modal techniques and advanced generative prior, SUPIR marks a significant advance in intelligent and realistic image restoration. As a pivotal catalyst within SUPIR, model scaling dramatically enhances its capabilities and demonstrates new potential for image restoration. Authors collect a dataset comprising 20 million high-resolution, high-quality images for model training, each enriched with descriptive text annotations. SUPIR provides the capability to restore images guided by textual prompts, broadening its application scope and potential

MonsterMMORPG DmitryRyumin splendiferousConiferous sombaba DonRichards timothyupai catastropiyush vipulg louisbrulenaudet

MonsterMMORPG distantquant cocodark StatsGary dotdotgod kevinpics mtasic85


Phi-3 Technical Report: A Highly Capable Language Model Locally on Your
  Phone (2404.14219)

Phi-3 Technical Report

A Highly Capable Language Model Locally on Your Phone

https://huggingface.co/papers/2404.14219

We introduce phi-3-mini, a 3.8 billion parameter language model trained on 3.3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3.5 (e.g., phi-3-mini achieves 69% on MMLU and 8.38 on MT-bench), despite being small enough to be deployed on a phone. The innovation lies entirely in our dataset for training, a scaled-up version of the one used for phi-2, composed of heavily filtered web data and synthetic data. The model is also further aligned for robustness, safety, and chat format. We also provide some initial parameter-scaling results with a 7B and 14B models trained for 4.8T tokens, called phi-3-small and phi-3-medium, both significantly more capable than phi-3-mini (e.g., respectively 75% and 78% on MMLU, and 8.7 and 8.9 on MT-bench).

himanshubeniwal wchai RahulSharma0 dotdotgod AtAndDev


Testing the Phi-3-mini 4k on HuggingChat. How well can it craft a tweet?

Not bad at all:
```
Excited to unveil phi-3-mini, a compact yet powerful 3.8B parameter model, outperforming giants like Mixtral & GPT-3.5 on benchmarks & safe for phones! *
#Al #Phi3 #LanguageModel #Techinnovation #Phi3Miniml
```
The models are here:
- Phi-3-Mini-4K-Instruct: https://huggingface.co/microsoft/Phi-3-mini-4k-instruct
-Phi-3-Mini-128K-Instruct: https://huggingface.co/microsoft/Phi-3-mini-128k-instruct

Try it out in Hugging Chat: https://huggingface.co/chat/models/microsoft/Phi-3-mini-4k-instruct


Happy to announce https://imgsys.org -- a sister project to Chatbot Arena by lmsys -- for comparing different text guided image generation models models. Try it natively on HuggingFace: https://huggingface.co/spaces/fal-ai/imgsys

clefourrier clem julien-c victor radames multimodalart badayvedat DmitryRyumin seyf1elislam lysandre ArthurZ cansa KingNish


Thrilled to introduce our Hyper-SD, offering hyper-fast⚡ and hyper-quality✨ text-to-image generation.
Our model achieves SINGLE-STEP inference on both SD1.5 and SDXL architecture without evident losses of aesthetics, styles and structures.
Project page: https://hyper-sd.github.io
HuggingFace repo: https://huggingface.co/ByteDance/Hyper-SD

renyuxi multimodalart DmitryRyumin Tonic YaTharThShaRma999 radames victor alielfilali01


All You need To Know About Phi-3 (Technical Report Walkthrough)

Summary of Summaries:
Phi-3-mini
- Architecture specs: decoder-only transformer, ModelSize: 3.8 billion
  parameters,  LongRope [ 128K Context length ], Vocab Size [ 32064 ],
  trained on 3.3 trillion tokens. at bfloat16.
- Rivals performance to larger models like Mixtral 8x7B and GPT-3.5,
  capable of running locally on a smartphone.
- Utilizes high quality training dataset heavily filtered from web data and
   llm-generated synthetic data.
- Can be quantized to 4-bits, occupying ≈ 1.8GB of memory.
- Ran natively on iPhone 14 with A16 Bionic chip with inference speed of up
   to 12 tokens per second.

Phi-3-small
- Architecture specs: Also decoder-only, 7B parameters, Vocab size [ 100352 ], default context length [ 8k ], Context Length: 8K, Hidden Dimension: 4096, Number of Heads and Layers: Follows 7B class structure.
- Uses tiktoken tokenizer (for enhanced multilingual tokenization)

Phi-3-medium:
-  Architecture specs: Also decoder-only, Hidden Dimension: 5120, Number of Heads: 40, Number of Layers: 40, Tokenization: Consistent with other models, Training on 4.8 trillion tokens.

Training Methodology:
- Focuses on high-quality training data deviating from standard scaling laws.
- The models undergo two-phase pre-training using a mix of web sources and synthetic data for general knowledge and logical reasoning skills.

Performance:
- Phi-3-mini achieves competitive scores on standard benchmarks like MMLU and MT-Bench, indicating strong reasoning capabilities.
- Higher variants show even better performance, suggesting effective scaling with increased model size.

Limitations:
- phi-3-mini: limited by its smaller size in tasks requiring extensive factual knowledge, primarily  supports English.
- phi-3-small limited multilingual support.

Hosting LLMs locally is a big win for OSS - private, secured inferencing on the go😎

LHPKAI heydrnik Tonic xiyuzhai Akhil2507 mrfakename Adriato MissKinziRoze victor rashakol19 AtAndDev youknowwhatAImean EddyGiusepe cuongtaongoc


Learn2Talk: 3D Talking Face Learns from 2D Talking Face (2404.12888)

🚀🎭🔥 New Research Alert (Avatars Collection)! 🔥👄🚀
📄 Title: Learn2Talk: 3D Talking Face Learns from 2D Talking Face

📝 Description: Learn2Talk is a framework that leverages expertise from 2D talking face methods to improve 3D talking face synthesis, focusing on lip synchronization and speech perception.

👥 Authors: Yixiang Zhuang et al.

🔗 Paper: https://huggingface.co/papers/2404.12888

🌐 Github Page: https://lkjkjoiuiu.github.io/Learn2Talk/

📚 More Papers: more cutting-edge research presented at other conferences in the https://huggingface.co/spaces/DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin

🚀 Added to the Avatars Collection: https://huggingface.co/collections/DmitryRyumin/avatars-65df37cdf81fec13d4dbac36

🔍 Keywords: #Learn2Talk #3DTalkingFace #SpeechDrivenFacialAnimation #LipSync #SpeechPerception #ComputerVision #ImageProcessing #DeepLearning

DmitryRyumin KvrParaskevi AlekseiPravdin alicegaz


Trained another version of llama3-8b-instruct which beats the base model. This time without losing too many points on gsm8k benchmark. Again, using AutoTrain 💥 pip install autotrain-advanced
Trained model: https://huggingface.co/abhishek/autotrain-llama3-orpo-v2

abhishek clem GAurav19 not-lain louisbrulenaudet JayGhiya


Why does Meta invest millions in Llama 3 and then makes it available for free? Here is Zuckerberg's explanation to investors in the Q3 2023 earnings call:

"The second part of our playbook is open source software infrastructure. Our long-standing strategy has been to build and open source general infrastructure while keeping our specific product implementations proprietary.

[...] First, open source software is typically safer and more secure, as well as more compute efficient to operate due to all the ongoing feedback, scrutiny, and development from the community. This is a big deal because safety is one of the most important issues in AI. Efficiency improvements and lowering the compute costs also benefit everyone including us.

Second, open source software often becomes an industry standard, and when companies standardize on building with our stack, that then becomes easier to integrate new innovations into our products. That’s subtle, but the ability to learn and improve quickly is a huge advantage and being an industry standard enables that.

Third, open source is hugely popular with developers and researchers. We know that people want to work on open systems that will be widely adopted, so this helps us recruit the best people at Meta, which is a very big deal for leading in any new technology area.

And again, we typically have unique data and build unique product integrations anyway, so providing infrastructure like Llama as open source doesn't reduce our main advantages. This is why our long-standing strategy has been to open source general infrastructure and why I expect it to continue to be the right approach for us going forward."

Fully earnings call transcript: https://s21.q4cdn.com/399680738/files/doc_financials/2023/q4/META-Q4-2023-Earnings-Call-Transcript.pdf

dippatel2506 clem Vinzou boapps louisbrulenaudet m-ric ppsingh


How Good Are Low-bit Quantized LLaMA3 Models? An Empirical Study (2404.14047)

We release an empirical study to showcase "How Good Are Low-bit Quantized hashtag#LLaMA3 🦙 Models" with existing LLM quantization techniques!

In this study, the performance of the low-bit LLaMA3 models (especially LLaMA3-70B) is impressively notable. 🚀 However, the results also exposed significant performance degradation issues faced by existing quantization techniques when dealing with LLaMA3, especially under ultra-low bit-width.

We hope this study can serve as a reference for the LLM quantization community and promote the emergence of stronger LLM quantization methods in the context of LLaMA3's release. More work is on the way...

https://huggingface.co/papers/2404.14047

https://huggingface.co/collections/LLMQ/llama3-quantization-66251258525135aeda16513c


ByteDance released new distillation technique Hyper-SD (https://huggingface.co/ByteDance/Hyper-SD)  for efficient image generation.
Here a few demos:

Official:
Hyper-SDXL-1Step-T2I https://huggingface.co/spaces/ByteDance/Hyper-SDXL-1Step-T2I

Hyper-SD15-Scribble https://huggingface.co/spaces/ByteDance/Hyper-SD15-Scribble

Unofficial Demos: InstantStyle + Hyper SD1.5 (not great but super fast) https://huggingface.co/spaces/radames/InstantStyle-Hyper-SD

InstantStyle + Hyper SDXL https://huggingface.co/spaces/radames/InstantStyle-Hyper-SDXL


I'm super happy to co-organize the (Mechanistic) Interpretability social at #ICLR2024 with @nikhil07prakash ! 🔍

If you plan to attend, help us make this meetup awesome by filling the form below! 😄

📅 Wed, May 8, 12:45-2:15 PM
🔗 RSVP & share your ideas here: https://forms.gle/FWap4KW2ikdntjfb8


Introduction to HelpingAI-3B-v2.1 and HelpingAI-3B-v2.2

These models are designed to engage in dialogues that are not only contextually relevant but also deeply empathetic, tailoring their responses to the emotional state and context of the user. With advanced emotional understanding capabilities, they can comfort, celebrate, and address complex feelings with a level of emotional nuance that is unparalleled in the AI domain.

Emotional Intelligence Capabilities

The core of these models lies in their emotional intelligence capabilities. They exhibit several key traits that enable them to respond emotionally resonant:

- Emotion recognition and validation: The models can identify and validate the emotions expressed by the user, ensuring that their responses are grounded in understanding.
- Empathetic perspective-taking: They are capable of taking the user's perspective, understanding their feelings, and responding with empathy.
- Generating emotionally supportive language: The models can generate language that is supportive and comforting, tailored to the user's emotional state.
- Contextual emotional attunement: They can adapt their communication style to match the emotional context of the conversation, ensuring relevance and appropriateness.
- Using appropriate tone, word choice, and emotional expression**: The models are adept at using the right tone, words, and emotional expressions to convey their messages effectively.

Examples of Emotionally Intelligent Responses

Let's take a look at how these models can adapt their communication style with emotional nuance:

- Comforting someone grieving: The models can offer empathetic responses, acknowledging the user's feelings and suggesting practical ways to cope with grief.
- Celebrating positive news: They can generate responses that are joyful and supportive, encouraging the user to enjoy their positive experiences.
https://huggingface.co/OEvortex/HelpingAI-3B-v2.1
https://huggingface.co/OEvortex/HelpingAI-3B-v2.2


InstantStyle works with the 2-step SDXL-Lightning distilled model, reducing generation time from ~20s to ~9s!

In a big related update, as of today, Diffusers@main supports InstantStyle. I'm looking forward to playing with it!

https://github.com/huggingface/diffusers/pull/7668

https://huggingface.co/spaces/radames/InstantStyle-SDXL-Lightning
https://huggingface.co/ByteDance/SDXL-Lightning


Unlocking the Power of locally running Llama-3 8B Model Agents with Chat-UI! 🔥🚀✨

I'm thrilled to share my hackathon-style side project:
1. Finetuning Llama-8B for function calling using PEFT QLoRA as the instruct Llama-3 model doesn't support this. The colab notebook for it is here: https://lnkd.in/ggJMzqh2. 🛠
2. Finetuned model along with the 4-bit quants here: https://lnkd.in/gNpFKY6V ✨
3. Clone Hugging Face https://lnkd.in/gKBKuUBQ and make it compatible for function calling by building upon the PR https://lnkd.in/gnqFuAd4 for my model and local inferencing usecase using Ollama. This was a steep learning curve wherein I stayed awake the whole night to get it working. 💪🏽
4. Above, I used SerpAPI for web browsing and Mongo DB Atlas free tier for persistence of conversations and assistant configs. 🔎
5. More work is required to switch between using tools and responding directly wherein I see the model breaks. 🧐

How cool is this wherein we are approaching experience akin to ChatGPT while using local hosted agent model running on your laptop! 💻

zarazi Kukedlc Abhaykoul AlekseiPravdin Tonic victor RedSparkie jaocseverywhere johnbendi

DmitryRyumin Tonic Nayer-ali med4u alleniver kpi parsapico johnbendi


Great in-depth Llama-3 tests from @wolfram, of the models from Meta of course but also @MaziyarPanahi @emozilla @turboderp :  https://huggingface.co/blog/wolfram/llm-comparison-test-llama-3

Spotted by @jack-kumar

wolfram fdaudens raincandy-u MaziyarPanahi julien-c SaylorTwift


I’m recently experimenting with the Flux-Ultra Realism and Real Anime LoRA models, using the Flux.1-dev model as the base. The model and its demo example are provided in the Flux LoRA DLC collections.📃

🥳Demo : 🔗 https://huggingface.co/spaces/prithivMLmods/FLUX-LoRA-DLC

🥳Model:
-  https://huggingface.co/prithivMLmods/Canopus-LoRA-Flux-UltraRealism-2.0
-  https://huggingface.co/prithivMLmods/Flux-Dev-Real-Anime-LoRA

🥳For more details, please visit the README.md of the Flux LoRA DLC Space & https://huggingface.co/collections/prithivMLmods/lora-space-collections-6714b72e0d49e1c97fbd6a32

Minhajameen Stopwolf AdinaY prithivMLmods ai4life44


🌐 Introducing Websim.ai User Projects Dataset - https://huggingface.co/datasets/nyuuzyou/websim

Dataset highlights:
- 137,452 user projects from Websim.ai, a service for creating small sites using Large Language Models (LLMs)
- Primarily in English, with potential for multilingual content in generated websites
- Each entry includes: project metadata, user information, and generated HTML content
- Contains detailed information about project revisions, site generation, and user interactions
- Data covers a wide range of user-generated website projects created through AI assistance
- Dedicated to the public domain under Creative Commons Zero (CC0) license

The dataset can be used for analyzing AI-assisted web development trends, studying user behavior in LLM-powered creative tools, and exploring the capabilities of language models in web design.


This is probably a very hot take, but here goes nothing.

With the incredibly accurate LoRAs we see emerge for high quality models like FLUX from services like fal.ai that offer training within single digit minutes, e.g. 2 min per 1000 iterations.

Why the hell are people publishing private LoRAs as public models?!
Take a look at this listing: https://huggingface.co/models?other=base_model:adapter:black-forest-labs%2FFLUX.1-dev&sort=created

I would expect that people that hold a HF account have some kind of forward thinking. Heck, do you really want to give anyone the power to create ultra realistic images of yourself?!

Didn't we learn anything from social media?
I am puzzled..


Last Week in Medical AI: Top LLM Research Papers/Models 🔥
🏅 (October 12 - October 19, 2024)

Medical LLM & Other Models:
- OLAPH: Factual Biomedical LLM QA
- LLMD: Interpreting Longitudinal Medical Records
- LifeGPT: Generative Transformer for Cells
- MedCare: Decoupled Clinical LLM Alignment
- Y-Mol: Biomedical LLM for Drug Development

Frameworks and Methodologies:
- MedINST: Biomedical Instructions Meta Dataset
- Democratizing Medical LLMs via Language Experts
- MCQG-SRefine: Iterative Question Generation
- Adaptive Medical Language Agents
- MeNTi: Medical LLM with Nested Tools

Medical LLM Applications:
- AGENTiGraph: LLM Chatbots with Private Data
- MMed-RAG: Multimodal Medical RAG System
- Medical Graph RAG: Safe LLM via Retrieval
- MedAide: Multi-Agent Medical LLM Collaboration
- Synthetic Clinical Trial Generation

Medical LLMs & Benchmarks:
- WorldMedQA-V: Multimodal Medical LLM Dataset
- HEALTH-PARIKSHA: RAG Models Evaluation
- Synthetic Data for Medical Vision-Language

Now you can watch and listen to the latest Medical AI papers daily on our YouTube and Spotify channels as well!

- Youtube: https://youtu.be/LROOjWXUgvg?si=s-nNDOSD3BrsHYjQ
- Spotify : https://open.spotify.com/episode/12xeN2vnOTRdDrHbWqhV6I?si=bd7c8d9fee8049fd

aaditya benhachem Joseph717171 shetumohanto


How the 🗣🏆 leaderboard of a merged TTS Arena with the 🤗 Spaces fork would look like. These results are somewhat unreliable as some models have not challenged the other in the list. And the original TTS Arena used only narration type sentences.


🚀  Announcement for the Lovely community! 🚀

Just launched the https://huggingface.co/spaces/zamal/DeepSeek-VL-1.3B-Chat on Hugging Face, and it's ready for YOU to explore! 💬🖼

This full-fledged model is perfect for advanced image and text interactions, with zero GPU required. The Deepseek VL-1.3B Chat typically needs around 8 GB of VRAM and storage of almost 4 GB, but now you can experience it hassle-free right on our space!

Want something lighter? We’ve also uploaded a 4 bit quantized version (just around 1GB!), available on my profile. Perfect for those with limited hardware. 🌍🔍

Come try it now and see what this model can do! 🚀✨


Triton nanoGPT now has a custom cross entropy loss kernel 🚀
Next: matmul, gradually overthrowing all major PyTorch ops:)

Simplified pseudo for parallel cross-entropy loss compute:
- init program: get pid, compute offsets, load targets.
- init row_max and row_sum.
- for-loop1 (find max logits): update row_max with max logits.
- for-loop2 (compute softmax and loss): compute row_sum, update loss.
- add log(row_sum) and store loss.

Code: https://github.com/Jaykef/ai-algorithms/blob/main/triton_nanoGPT.ipynb


Looks like @Meta thinks we forgot they created PyTorch, so now they've open-sourced Lingua, a powerful and flexible library for training and inferencing large language models.

Things that stand out:

- Architecture: Pure PyTorch `nn.Module` implementation for easy customization.

- Checkpointing: Uses the new PyTorch distributed saving method (.distcp format) for flexible model reloading across different GPU configurations.

- Configuration: Utilizes data classes and YAML files for intuitive setup and modification.

- Profiling: Integrates with xFormers' profiler for automatic MFU and HFU calculation, plus memory profiling.

- Slurm Integration: Includes `stool.py` for seamless job launching on Slurm clusters.

Some results from @Meta to show off:

- 1B parameter models trained on 60B tokens achieve strong performance across various NLP tasks.

- 7B parameter Mamba model (trained on 200B tokens) shows competitive results with Llama 7B on benchmarks like ARC, MMLU, and BBH.

If you're working on LLM research or looking to experiment with cutting-edge language model architectures, Lingua is definitely worth exploring.


Interested in performing inference with an ONNX model?⚡

The Optimum docs about model inference with ONNX Runtime is now much clearer and simpler!

You want to deploy your favorite model on the hub but you don't know how to export it to the ONNX format? You can do it in one line of code as follows:
```py
from optimum.onnxruntime import ORTModelForSequenceClassification

# Load the model from the hub and export it to the ONNX format
model_id = "distilbert-base-uncased-finetuned-sst-2-english"
model = ORTModelForSequenceClassification.from_pretrained(model_id, export=True)
```
Check out the whole guide 👉 https://huggingface.co/docs/optimum/onnxruntime/usage_guides/models

John6666 adeiza-x louisbrulenaudet Aurelien-Morgan itsbrex


Enabling Natural Zero-Shot Prompting on Encoder Models via
  Statement-Tuning (2404.12897)

Prompting BERT!

Zero-shot learning ability is the hottest thing about causal LLMs. You don't need to finetune causal LLMs on each specific task. Instead, you can use prompting and get a decent performance on unseen tasks.

Unfortunately, autoencoding LLMs -  like our dear friend BERT 🙋♂- lack this ability and you need a task-specific head for different tasks. But what if you could prompt all the BERTs in the world?!

🥁 Introducing Statement-Tuning 🥁

Now hold your horses! don't go full-LLama on it yet. Using this finetuning approach, we can get zero-shot performance from encoders by turning a problem into a yes/no problem. Binary classification all the way down!
For example, a single entailment problem will be decomposed into 3 yes/no questions.

This is still not super useful. But I like works that try to make a little more space for encoders in the current autoregressive era!

Check the paper if interested: https://huggingface.co/papers/2404.12897

radames KingNish Unggi MarinaraSpaghetti


Idefics2 is trained mostly on OBELICS, our open interleaved image-text document dataset.

Training on interleaved data is crucial to reaching high performance on VQA tasks, taking an arbitrary number of images as input, and doing in-context learning.

Dataset: https://huggingface.co/datasets/HuggingFaceM4/OBELICS
Nomic visualization: https://atlas.nomic.ai/map/f2fba2aa-3647-4f49-a0f3-9347daeee499/ee4a84bd-f125-4bcc-a683-1b4e231cb10f
Link to OBELICS thread: https://twitter.com/HugoLaurencon/status/1694005892839006301


Please... feed this Llama some Sauerkraut! 🍲

Said and done. Here it is. Our Sauerkraut Version of the strong Llama3-8b by Meta. Released from HANNOVER MESSE, just in front of meta booth.
https://huggingface.co/VAGOsolutions/Llama-3-SauerkrautLM-8b-Instruct

According to benchmarks (LM-Evaluation-Harness 0.4.2), our #SauerkrautLM Dataset and fine-tuning pipeline improved the Model noticeably (AVG = 74,57), especially Reasoning and Common Sense capabilities.

Again we provide some more detail on the whole process:
✅ Original model: Llama-3-8b-Instruct
✅ Training Duration: 12 hours
✅ Training procedure: 2-staged DPO
✅ Trained data: 70k (first stage) and 20k (second stage)
✅ GPU: 4x RTX6000 ADA
✅ New model: Llama-3-SauerkrautLM-8b-Instruct
✅ Total training costs: 54,72 Dollar 💴 - RunPod FTW (excluding synthesizing data, curating data, benchmarks, error handling, testing)

See our model card on Hugging Face for more details: https://huggingface.co/VAGOsolutions/Llama-3-SauerkrautLM-8b-Instruct

There will be more details on benchmarks during the next days.

Xorg2024 clem DaryoushV quyettv fynnkroeger jdjayakaran2410


Meta Llama 3 70B landed on the Leaderboard at the 11th position: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard


"Is this a bug? I'm able to post even though they didn't approve my request."


**Release Announcement: gliner_multi_pii-v1**

I am pleased to announce the release of gliner_multi_pii-v1, a model developed for recognizing a wide range of Personally Identifiable Information (PII). This model is the result of fine-tuning the urchade/gliner_multi-v2.1 on synthetic dataset (urchade/synthetic-pii-ner-mistral-v1).

**Model Features:**
- Capable of identifying multiple PII types including addresses, passport numbers, emails, social security numbers, and more.
- Designed to assist with data protection and compliance across various domains.
- Multilingual (English, French, Spanish, German, Italian, Portugese)

Link: https://huggingface.co/urchade/gliner_multi_pii-v1

```python
from gliner import GLiNER

model = GLiNER.from_pretrained("urchade/gliner_multi_pii-v1")

text = """
Harilala Rasoanaivo, un homme d'affaires local d'Antananarivo, a enregistré une nouvelle société nommée "Rasoanaivo Enterprises" au Lot II M 92 Antohomadinika. Son numéro est le +261 32 22 345 67, et son adresse électronique est harilala.rasoanaivo@telma.mg. Il a fourni son numéro de sécu 501-02-1234 pour l'enregistrement.
"""

labels = ["work", "booking number", "personally identifiable information", "driver licence", "person",  "address", "company",  "email", "passport number", "Social Security Number", "phone number"]
entities = model.predict_entities(text, labels)

for entity in entities:
    print(entity["text"], "=>", entity["label"])
```

```
Harilala Rasoanaivo => person
Rasoanaivo Enterprises => company
Lot II M 92 Antohomadinika => full address
+261 32 22 345 67 => phone number
harilala.rasoanaivo@telma.mg => email
501-02-1234 => Social Security Number
```

bfuzzy1 mrdbourke leslyarun Tatvajsh ksquarekumar manu xclyfe bitsTobyte timpearce


ChemLLM: A Chemical Large Language Model (2402.06852)

ChemLLM datasets is all open source now!
https://huggingface.co/papers/2402.06852
700K of SFT Dataset, ChemData700K For Chemistry of LLM!
https://huggingface.co/datasets/AI4Chem/ChemData700K
10K of DPO Dataset, ChemPref-10K, both English and Chinese!
https://huggingface.co/datasets/AI4Chem/ChemPref-DPO-for-Chemistry-data-en
https://huggingface.co/datasets/AI4Chem/ChemPref-DPO-for-Chemistry-data-cn
ChemBench-4K of 4100 high-quality single-choice benchmark for nine core Chemistry tasks!
https://huggingface.co/datasets/AI4Chem/ChemBench4K
C-MHChem, 600 real test questions written and checked manually, from 25 years of Chinese National Middle school chemistry Test!
https://huggingface.co/datasets/AI4Chem/C-MHChem-Benchmark-Chinese-Middle-high-school-Chemistry-Test
All hail to Open-source community!🤗

DmitryRyumin qq8933 ajibawa-2023 clem mmhamdy Tonic DRRRRDFFD tiandatong


🔥  **transformers**  4.40.0 is out 🔥

⚙ you can now push your custom pipelines easily to 🤗, allowing people to easily use your model in a more friendly, unified way.

🤓 I already updated my blog to match the new feature https://huggingface.co/blog/not-lain/custom-architectures-with-huggingface.

📃A list of some repos that have custom pipelines :
* https://huggingface.co/briaai/RMBG-1.4
* https://huggingface.co/p1atdev/siglip-tagger-test-3
* https://huggingface.co/sgugger/test-dynamic-pipeline

DmitryRyumin bfuzzy1 zolicsaki Tonic louisbrulenaudet


#TPU #PyTorch #Jax

When You're trying to use PyTorch or Jax on TPU,

for v2/v3/v4:
  use tpu-ubuntu2204-base

for v5p:
  use v2-alpha-tpuv5

for v5e:
  use v2-alpha-tpuv5-lite

You must use these base images for the system to 'boot'.

Previously used tpu-vm-v4-pt-1.13 images might seem to start the VM, but SSH connections do not work.

I thought it was a firewall issue and spent a lot of time on it before realizing it was a problem with the boot image 🥲

https://cloud.google.com/tpu/docs/runtimes#pytorch_and_jax

deepkyu maywell devngho byoussef Taekyoon


Meta's new LLama-3 (https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) is an extremely capable model out of the box for coding related tasks. It is the first model that we have seen that beats GPT-4 on Static-Analysis-Eval - https://huggingface.co/datasets/patched-codes/static-analysis-eval.

codelion ArchieX25 AayushShah victor clem g-ronimo Tonic


PersonaLLM: Investigating the Ability of Large Language Models to
  Express Personality Traits (2305.02547)

😀🤓😎 New Research Alert - NAACL 2024 (Big Five Personality Traits Collection)! 😎😉😤
📄 Title: PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits 💬

📝 Description: This research examines the ability of LLMs to express personality traits and finds that LLMs can generate content consistent with assigned personality profiles and that humans can recognize certain traits with up to 80% accuracy. However, accuracy drops significantly when annotators are aware that the content was generated by an AI.

👥 Authors: Hang Jiang et al.

📅 Conference: NAACL, June 16–21, 2024 | Mexico City, Mexico 🇲🇽

🔗 Paper: https://huggingface.co/papers/2305.02547

📁 Repository: https://github.com/hjian42/PersonaLLM

🚀 Added to the Big Five Personality Traits Collection: https://huggingface.co/collections/DmitryRyumin/big-five-personality-traits-661fb545292ab3d12a5a4890

🔥🔝 See also OCEAN-AI - https://huggingface.co/spaces/ElenaRyumina/OCEANAI (App, co-authored by @DmitryRyumin) 😉

📚 More Papers: more cutting-edge research presented at other conferences in the https://huggingface.co/spaces/DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin

🔍 Keywords: #PersonaLLM #OCEANAI #BigFive #PersonalityTraits #PersonalityAnalysis #Chatbots #LLMs #NAACL2024 #DeepLearning #Innovation


Open Models vs. Closed APIs for Software Engineers
-----------------------------------------------------------------------

If you're an ML researcher / scientist, you probably don't need much convincing to use open models instead of closed APIs -- open models give you reproducibility and let you deeply investigate the model's behavior.

But what if you are a software engineer building products on top of LLMs? I'd argue that open models are a much better option even if you are using them as APIs. For at least 3 reasons:

1) The most obvious reason is reliability of your product. Relying on a closed API means that your product has a single point-of-failure. On the other hand, there are at least 7 different API providers that offer Llama3 70B already. As well as libraries that abstract on top of these API providers so that you can make a single request that goes to different API providers depending on availability / latency.

2) Another benefit is eventual consistency going local. If your product takes off, it will be more economical and lower latency to have a dedicated inference endpoint running on your VPC than to call external APIs. If you've started with an open-source model, you can always deploy the same model locally. You don't need to modify prompts or change any surrounding logic to get consistent behavior. Minimize your technical debt from the beginning.

3) Finally, open models give you much more flexibility. Even if you keep using APIs, you might want to tradeoff latency vs. cost, or use APIs that support batches of inputs, etc. Because different API providers have different infrastructure, you can use the API provider that makes the most sense for your product -- or you can even use multiple API providers for different users (free vs. paid) or different parts of your product (priority features vs. nice-to-haves)

DmitryRyumin ameerazam08 MAsad789565 AboAhmed901 dilber toast224 clem VictorSanh julien-c osanseviero yjernite jeffboudier GiorgioDiSalvo meton aceeee helezabi

clem clefourrier yjernite m-conrad-202 louisbrulenaudet aceeee Bagus


Already almost 1,000 llama3 model variations have been shared publicly on HF (many more in private use at companies): https://huggingface.co/models?p=5&sort=trending&search=llama3.

Everyone should fine-tune their own models for their use-cases, languages, industry, infra constraints,...

10,000 llama3 variants by the end of next week?

ijohn07 qnixsynapse Aeio DmitryRyumin Kathakali tomaarsen qubvel-hf rksiitd victor Jaward radames shiv2050 praveenpankaj Tonic rasonyang

yo DmitryRyumin alielfilali01 sepal tomaarsen fbjr raincandy-u radames Tonic louisbrulenaudet