Accessibility of Huggingface's OpenLLMLeaderboard Benchmark Test Sets
Is the new Huggingface OpenLLMLeaderboard’s benchmark test sets accessible to the public? If so, what measures are in place to prevent dataset misuse for fine-tuning and leaderboard manipulation?
2024-07-17T13:03:02Z
Hi@bahgatYes, the Huggingface OpenLLMLeaderboard’s benchmark test sets are public. To prevent misuse, they have data usage agreements, activity monitoring, and regular audits.Have you seen similar measures on other benchmark platforms?
2024-07-17T13:45:08Z
Thanks@LLUMOAIfor your reply. I’m not familiar with measures on other benchmark platforms, so I can’t make comparisons. However, I’m interested in learning more about the specific measures you mentioned for the Huggingface OpenLLMLeaderboard.Could you please provide a source where Huggingface has officially stated these measures? I’d like to see more details about:The data usage agreements - What exactly do they entail?Activity monitoring - How is this implemented?Regular audits - What do these audits involve and how often are they conducted?
2024-07-23T14:04:39Z
Hi@bahgatFor details on Hugging Face OpenLLMLeaderboard measures:Data Usage Agreements:See Terms of Serviceand Data Policies.Activity Monitoring: Check theOpenLLMLeaderboarddocs.Regular Audits: Refer to theirTransparency Report.Hope this helps!