HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs
Paper
โข
2412.18925
โข
Published
โข
82
ASAS AI
community
AI & ML interests
Arabic NLP, CV, AI Safety and ethics
Recent Activity
View all activity
asas-ai's activity
Benyouย
authored
a
paper
4 days ago
mmhamdyย
authored
a
paper
4 days ago
alielfilali01ย
posted
an
update
5 days ago
Post
1673
~75% on the challenging GPQA with only 40M parameters ๐ฅ๐ฅณ
GREAT ACHIEVEMENT ! Or is it ?
This new Work, "Data Laundering: Artificially Boosting Benchmark Results through Knowledge Distillation", take out the mystery about many models i personally suspected their results. Speacially on leaderboards other than the english one, Like the Open Arabic LLM Leaderbaord OALL/Open-Arabic-LLM-Leaderboard.
The authors of this work, first started by training a model on the GPQA data, which, unsurprisingly, led to the model achieving 100% performance.
Afterward, they trained what they referred to as a 'legitimate' model on legitimate data (MedMCQA). However, they introduced a distillation loss from the earlier, 'cheated' model.
What they discovered was fascinating: the knowledge of GPQA leaked through this distillation loss, even though the legitimate model was never explicitly trained on GPQA during this stage.
This raises important questions about the careful use of distillation in model training, especially when the training data is opaque. As they demonstrated, itโs apparently possible to (intentionally or unintentionally) leak test data through this method.
Find out more: Data Laundering: Artificially Boosting Benchmark Results through Knowledge Distillation (2412.15255)
GREAT ACHIEVEMENT ! Or is it ?
This new Work, "Data Laundering: Artificially Boosting Benchmark Results through Knowledge Distillation", take out the mystery about many models i personally suspected their results. Speacially on leaderboards other than the english one, Like the Open Arabic LLM Leaderbaord OALL/Open-Arabic-LLM-Leaderboard.
The authors of this work, first started by training a model on the GPQA data, which, unsurprisingly, led to the model achieving 100% performance.
Afterward, they trained what they referred to as a 'legitimate' model on legitimate data (MedMCQA). However, they introduced a distillation loss from the earlier, 'cheated' model.
What they discovered was fascinating: the knowledge of GPQA leaked through this distillation loss, even though the legitimate model was never explicitly trained on GPQA during this stage.
This raises important questions about the careful use of distillation in model training, especially when the training data is opaque. As they demonstrated, itโs apparently possible to (intentionally or unintentionally) leak test data through this method.
Find out more: Data Laundering: Artificially Boosting Benchmark Results through Knowledge Distillation (2412.15255)
- 1 reply
alielfilali01ย
posted
an
update
22 days ago
Post
3383
Unpopular opinion: Open Source takes courage to do !
Not everyone is brave enough to release what they have done (the way they've done it) to the wild to be judged !
It really requires a high level of "knowing wth are you doing" ! It's kind of a super power !
Cheers to the heroes here who see this!
Not everyone is brave enough to release what they have done (the way they've done it) to the wild to be judged !
It really requires a high level of "knowing wth are you doing" ! It's kind of a super power !
Cheers to the heroes here who see this!
-
- 3 replies
alielfilali01ย
posted
an
update
26 days ago
Post
1505
Apparently i forgot to put this here !
Well, this is a bit late but consider given our recent blog a read if you are interested in Evaluation.
You don't have to be into Arabic NLP in order to read it, the main contribution we are introducing is a new evaluation measure for NLG. We made the fisrt application of this measure on Arabic for now and we will be working with colleagues from the community to expand it to other languages.
Blog:
Rethinking LLM Evaluation with 3C3H: AraGen Benchmark and Leaderboard
https://huggingface.co/blog/leaderboard-3c3h-aragen
Space:
inceptionai/AraGen-Leaderboard
Give it a read and let me know your thoughts ๐ค
Well, this is a bit late but consider given our recent blog a read if you are interested in Evaluation.
You don't have to be into Arabic NLP in order to read it, the main contribution we are introducing is a new evaluation measure for NLG. We made the fisrt application of this measure on Arabic for now and we will be working with colleagues from the community to expand it to other languages.
Blog:
Rethinking LLM Evaluation with 3C3H: AraGen Benchmark and Leaderboard
https://huggingface.co/blog/leaderboard-3c3h-aragen
Space:
inceptionai/AraGen-Leaderboard
Give it a read and let me know your thoughts ๐ค
mmhamdyย
authored
a
paper
30 days ago
Benyouย
authored
a
paper
about 1 month ago
alielfilali01ย
posted
an
update
about 2 months ago
Post
2185
Unpopular opinion : o1-preview is more stupid than 4o and Qwen2.5-72B-Instruct in extremely underrated !
- 2 replies
Benyouย
authored
a
paper
2 months ago
alielfilali01ย
posted
an
update
2 months ago
Post
1702
I feel like this incredible resource hasn't gotten the attention it deserves in the community!
@clefourrier and generally the HuggingFace evaluation team put together a fantastic guidebook covering a lot about ๐๐ฉ๐๐๐จ๐๐ง๐๐ข๐ก from basics to advanced tips.
link : https://github.com/huggingface/evaluation-guidebook
I havenโt finished it yet, but i'am enjoying every piece of it so far. Huge thanks @clefourrier and the team for this invaluable resource !
@clefourrier and generally the HuggingFace evaluation team put together a fantastic guidebook covering a lot about ๐๐ฉ๐๐๐จ๐๐ง๐๐ข๐ก from basics to advanced tips.
link : https://github.com/huggingface/evaluation-guidebook
I havenโt finished it yet, but i'am enjoying every piece of it so far. Huge thanks @clefourrier and the team for this invaluable resource !
-
-
- 3 replies
Benyouย
authored
a
paper
3 months ago
alielfilali01ย
posted
an
update
3 months ago
Post
1829
Why nobdoy is talking about the new training corpus released by MBZUAI today.
TxT360 is +15 Trillion tokens corpus outperforming FineWeb on several metrics. Ablation studies were done up to 1T tokens.
Read blog here : LLM360/TxT360
Dataset : LLM360/TxT360
TxT360 is +15 Trillion tokens corpus outperforming FineWeb on several metrics. Ablation studies were done up to 1T tokens.
Read blog here : LLM360/TxT360
Dataset : LLM360/TxT360
-
- 2 replies
Post
1880
๐ Evaluating Long Context #1: Long Range Arena (LRA)
Accurately evaluating how well language models handle long contexts is crucial, but it's also quite challenging to do well. In this series of posts, we're going to examine the various benchmarks that were proposed to assess long context understanding, starting with Long Range Arens (LRA)
Introduced in 2020, Long Range Arens (LRA) is one of the earliest benchmarks designed to tackle the challenge of long context evaluation.
๐ Key Features of LRA
1๏ธโฃ Diverse Tasks: The LRA benchmark consists of a suite of tasks designed to evaluate model performance on long sequences ranging from 1,000 to 16,000 tokens. These tasks encompass different data types and modalities: Text, Natural and Synthetic Images, and Mathematical Expressions.
2๏ธโฃ Synthetic and Real-world Tasks: LRA is comprised of both synthetic probing tasks and real-world tasks.
3๏ธโฃ Open-Source and Extensible: Implemented in Python using Jax and Flax, the LRA benchmark code is publicly available, making it easy to extend.
๐ Tasks
1๏ธโฃ Long ListOps
2๏ธโฃ Byte-level Text Classification and Document Retrieval
3๏ธโฃ Image Classification
4๏ธโฃ Pathfinder and Pathfinder-X (Long-range spatial dependency)
๐จโ๐ป Long Range Arena (LRA) Github Repository: https://github.com/google-research/long-range-arena
๐ Long Range Arena (LRA) paper: Long Range Arena: A Benchmark for Efficient Transformers (2011.04006)
Accurately evaluating how well language models handle long contexts is crucial, but it's also quite challenging to do well. In this series of posts, we're going to examine the various benchmarks that were proposed to assess long context understanding, starting with Long Range Arens (LRA)
Introduced in 2020, Long Range Arens (LRA) is one of the earliest benchmarks designed to tackle the challenge of long context evaluation.
๐ Key Features of LRA
1๏ธโฃ Diverse Tasks: The LRA benchmark consists of a suite of tasks designed to evaluate model performance on long sequences ranging from 1,000 to 16,000 tokens. These tasks encompass different data types and modalities: Text, Natural and Synthetic Images, and Mathematical Expressions.
2๏ธโฃ Synthetic and Real-world Tasks: LRA is comprised of both synthetic probing tasks and real-world tasks.
3๏ธโฃ Open-Source and Extensible: Implemented in Python using Jax and Flax, the LRA benchmark code is publicly available, making it easy to extend.
๐ Tasks
1๏ธโฃ Long ListOps
2๏ธโฃ Byte-level Text Classification and Document Retrieval
3๏ธโฃ Image Classification
4๏ธโฃ Pathfinder and Pathfinder-X (Long-range spatial dependency)
๐จโ๐ป Long Range Arena (LRA) Github Repository: https://github.com/google-research/long-range-arena
๐ Long Range Arena (LRA) paper: Long Range Arena: A Benchmark for Efficient Transformers (2011.04006)
alielfilali01ย
posted
an
update
3 months ago
Post
2571
Don't you think we should add a tag "Evaluation" for datasets that are meant to be benchmarks and not for training ?
At least, when someone is collecting a group of datasets from an organization or let's say the whole hub can filter based on that tag and avoid somehow contaminating their "training" data.
At least, when someone is collecting a group of datasets from an organization or let's say the whole hub can filter based on that tag and avoid somehow contaminating their "training" data.
alielfilali01ย
posted
an
update
3 months ago
Post
873
We need a fork feature for models and datasets similar to "Duplicate this space" in spaces ! Don't you think ?
Sometimes you just want to save something in your profile privately and work on it later without the hassle of "load_.../push_to_hub" in a code file.
I know this is super lazy ๐ But it is what it is ...
tag : @victor
Sometimes you just want to save something in your profile privately and work on it later without the hassle of "load_.../push_to_hub" in a code file.
I know this is super lazy ๐ But it is what it is ...
tag : @victor
-
-
- 5 replies
Benyouย
authored
5
papers
3 months ago
Natural Language Reasoning, A Survey
Paper
โข
2303.14725
โข
Published
โข
1
AceGPT, Localizing Large Language Models in Arabic
Paper
โข
2309.12053
โข
Published
Phoenix: Democratizing ChatGPT across Languages
Paper
โข
2304.10453
โข
Published
Huatuo-26M, a Large-scale Chinese Medical QA Dataset
Paper
โข
2305.01526
โข
Published
Word Grounded Graph Convolutional Network
Paper
โข
2305.06434
โข
Published