ArtificialAnalysis/hf-assets
Viewer
ā¢
Updated
ā¢
7
ā¢
1.38k
Artificial Analysis
company
AI & ML interests
None defined yet.
Recent Activity
View all activity
ArtificialAnalysis's activity
mhillsmithĀ
updated
2
datasets
15 days ago
georgewritescodeĀ
updated
a
dataset
15 days ago
When include new models like SORA and HunYuan?
2
#2 opened 24 days ago
by
RaphaelLiu
georgewritescodeĀ
updated
a
Space
about 1 month ago
georgewritescodeĀ
updated
2
Spaces
7 months ago
georgewritescodeĀ
posted
an
update
8 months ago
Post
1020
Visualization of GPT-4o breaking away from the quality & speed trade-off curve the LLMs have followed thus far āļø
Key GPT-4o takeaways
ā£ GPT-4o not only offers the highest quality, it also sits amongst the fastest LLMs
ā£ For those with speed/latency-sensitive use cases, where previously Claude 3 Haiku or Mixtral 8x7b were leaders, GPT-4o is now a compelling option (though significantly more expensive)
ā£ Previously Groq was the only provider to break from the curve using its own LPU chips. OpenAI has done it on Nvidia hardware (one can imagine the potential for GPT-4o on Groq)
š How did they do it? Will follow up with more analysis on this but potential approaches include a very large but sparse MoE model (similar to Snowflake's Arctic) and improvements in data quality (likely to have driven much of Llama 3's impressive quality relative to parameter count)
Notes: Throughput represents the median across providers over the last 14 days of measurements (8x per day)
Data is present on our HF leaderboard: ArtificialAnalysis/LLM-Performance-Leaderboard and graphs present on our website
Key GPT-4o takeaways
ā£ GPT-4o not only offers the highest quality, it also sits amongst the fastest LLMs
ā£ For those with speed/latency-sensitive use cases, where previously Claude 3 Haiku or Mixtral 8x7b were leaders, GPT-4o is now a compelling option (though significantly more expensive)
ā£ Previously Groq was the only provider to break from the curve using its own LPU chips. OpenAI has done it on Nvidia hardware (one can imagine the potential for GPT-4o on Groq)
š How did they do it? Will follow up with more analysis on this but potential approaches include a very large but sparse MoE model (similar to Snowflake's Arctic) and improvements in data quality (likely to have driven much of Llama 3's impressive quality relative to parameter count)
Notes: Throughput represents the median across providers over the last 14 days of measurements (8x per day)
Data is present on our HF leaderboard: ArtificialAnalysis/LLM-Performance-Leaderboard and graphs present on our website
- 1 reply
georgewritescodeĀ
posted
an
update
8 months ago
Post
2327
Excited to bring our benchmarking leaderboard of >100 LLM API endpoints to HF!
Speed and price are often just as important as quality when building applications with LLMs. We bring together all the data you need to consider all three when you need to pick a model and API provider.
Coverage:
ā£ Quality (Index of evals, MMLU, Chatbot Arena, HumanEval, MT-Bench)
ā£ Throughput (tokens/s: median, P5, P25, P75, P95)
ā£ Latency (TTFT: median, P5, P25, P75, P95)
ā£ Context window
ā£ OpenAI library compatibility
Link to Space: ArtificialAnalysis/LLM-Performance-Leaderboard
Blog post: https://huggingface.co/blog/leaderboard-artificial-analysis
Speed and price are often just as important as quality when building applications with LLMs. We bring together all the data you need to consider all three when you need to pick a model and API provider.
Coverage:
ā£ Quality (Index of evals, MMLU, Chatbot Arena, HumanEval, MT-Bench)
ā£ Throughput (tokens/s: median, P5, P25, P75, P95)
ā£ Latency (TTFT: median, P5, P25, P75, P95)
ā£ Context window
ā£ OpenAI library compatibility
Link to Space: ArtificialAnalysis/LLM-Performance-Leaderboard
Blog post: https://huggingface.co/blog/leaderboard-artificial-analysis