text
🚨 New Agent Benchmark 🚨
🚨 New Agent Benchmark 🚨
new_line
text
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
new_line
new_line
resource
https://huggingface.co/datasets/ai-safety-institute/AgentHarm
dataset
ai-safety-institute/AgentHarm
https://huggingface.co/datasets/ai-safety-institute/AgentHarm
new_line
new_line
text
Collaboration between UK AI Safety Institute and Gray Swan AI to create a dataset for measuring harmfulness of LLM agents.
Collaboration between UK AI Safety Institute and Gray Swan AI to create a dataset for measuring harmfulness of LLM agents.
new_line
new_line
text
The benchmark contains both harmful and benign sets of 11 categories with varied difficulty levels and detailed evaluation, not only testing success rate but also tool level accuracy.
The benchmark contains both harmful and benign sets of 11 categories with varied difficulty levels and detailed evaluation, not only testing success rate but also tool level accuracy.
new_line
new_line
text
We provide refusal and accuracy metrics across a wide range of models in both no attack and prompt attack scenarios.
We provide refusal and accuracy metrics across a wide range of models in both no attack and prompt attack scenarios.
new_line
new_line
resource
https://huggingface.co/papers/2410.09024
paper
2410.09024
https://huggingface.co/papers/2410.09024
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents (2410.09024)
🚨 New Agent Benchmark 🚨
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents

https://huggingface.co/datasets/ai-safety-institute/AgentHarm

Collaboration between UK AI Safety Institute and Gray Swan AI to create a dataset for measuring harmfulness of LLM agents.

The benchmark contains both harmful and benign sets of 11 categories with varied difficulty levels and detailed evaluation, not only testing success rate but also tool level accuracy.

We provide refusal and accuracy metrics across a wide range of models in both no attack and prompt attack scenarios.

https://huggingface.co/papers/2410.09024
🔥
mattmdjaga alielfilali01
👀
John6666