README.md

---
license: other
license_name: qwen-research
license_link: https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct/blob/main/LICENSE
language:
- en
base_model:
- Qwen/Qwen2.5-Coder-3B-Instruct
pipeline_tag: text-generation
library_name: transformers
tags:
- code
- chat
- qwen
- qwen-coder
- agent
---

# Dria-Agent-α-3B

## Introduction

***Dria-Agent-α*** are series of large language models trained on top of the [Qwen2.5-Coder](https://huggingface.co/collections/Qwen/qwen25-coder-66eaa22e6f99801bf65b0c2f) series, specifically on top of the [Qwen/Qwen2.5-Coder-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct) and [Qwen/Qwen2.5-Coder-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct) models to be used in agentic applications. These models are the first instalment of agent-focused LLMs (hence the **α** in the naming) we hope to improve with better and more elaborate techniques in subsequent releases.

Dria-Agent-α employs ***Pythonic function calling***, which is LLMs using blocks of Python code to interact with provided tools and output actions. This method was inspired by many previous work, including but not limited to [DynaSaur](https://arxiv.org/pdf/2411.01747), [RLEF](https://arxiv.org/pdf/2410.02089), [ADAS](https://arxiv.org/pdf/2408.08435) and [CAMEL](https://arxiv.org/pdf/2303.17760). This way of function calling has a few advantages over traditional JSON-based function calling methods:

1. **One-shot Parallel Multiple Function Calls:** The model can can utilise many synchronous processes in one chat turn to arrive to a solution, which would require other function calling models multiple turns of conversation.
2. **Free-form Reasoning and Actions:** The model provides reasoning traces freely in natural language and the actions in between \`\`\`python \`\`\` blocks, as it already tends to do without special prompting or tuning. This tries to mitigate the possible performance loss caused by imposing specific formats on LLM outputs discussed in [Let Me Speak Freely?](https://arxiv.org/pdf/2408.02442)
3. **On-the-fly Complex Solution Generation:** The solution provided by the model is essentially a Python program with the exclusion of some "risky" builtins like `exec`, `eval` and `compile` (see full list in **Quickstart** below). This enables the model to implement custom complex logic with conditionals and synchronous pipelines (using the output of one function in the next function's arguments) which would not be possible with the current JSON-based function calling methods (as far as we know).

## Quickstart

````python
import json
from typing import Any, Dict, List
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "driaforall/Dria-Agent-a-3B"
model = AutoModelForCausalLM.from_pretrained(
    model_name, device_map="auto", torch_dtype="auto", trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Please use our provided prompt for best performance
SYSTEM_PROMPT = """
You are an expert AI assistant that specializes in providing Python code to solve the task/problem at hand provided by the user.

You can use Python code freely, including the following available functions:

<|functions_schema|>
{{functions_schema}}
<|end_functions_schema|>

The following dangerous builtins are restricted for security:
- exec
- eval
- execfile
- compile
- importlib
- input
- exit

Think step by step and provide your reasoning, outside of the function calls.
You can write Python code and use the available functions. Provide all your python code in a SINGLE markdown code block like the following:

```python
result = example_function(arg1, "string")
result2 = example_function2(result, arg2)
```

DO NOT use print() statements AT ALL. Avoid mutating variables whenever possible.
""".strip()


get_sample_data = """
def check_availability(day: str, start_time: str, end_time: str) -> bool:
    \"\"\" 
    Check if a time slot is available on a given day.

    Args:
    - day: The day to check in YYYY-MM-DD format
    - start_time: Start time in HH:MM format
    - end_time: End time in HH:MM format

    Returns:
    - True if slot is available, False otherwise
    \"\"\"
    pass

def make_appointment(day: str, start_time: str, end_time: str) -> dict:
    \"\"\" 
    Make an appointment for a given time slot.

    Args:
    - day: The day to make appointment in YYYY-MM-DD format
    - start_time: Start time in HH:MM format
    - end_time: End time in HH:MM format
    - title: The title of the appointment

    Returns:
    - A dictionary with the appointment details and if it's made or not.
        dict keys:
            - day (str): The day the appointment is on, in YYYY-MM-DD format
            - start_time (str): Start time in HH:MM format
            - end_time (str): End time in HH:MM format
            - appointment_made (bool): Whether the appointment is successfully made or not. 
    \"\"\"
    pass

def add_to_reminders(reminder_text: str) -> bool:
    \"\"\" 
    Add a text to reminders.

    Args: 
    - reminder_text: The text to add to reminders

    Returns:
    - Whether the reminder was successfully created or not.
    \"\"\"
    pass
"""

# Helper function to create the system prompt for our model
def format_prompt(tools: str):
    return SYSTEM_PROMPT.format(functions_schema=tools)

system_prompt = SYSTEM_PROMPT.replace("{{functions_schema}}", get_sample_data)

USER_QUERY = """
Can you check if I have tomorrow 10:00-12:00 available and make an appointment for a meeting 
with my thesis supervisor if so? If you made the appointment, please add it to my reminders.
"""

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": USER_QUERY},
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=2048
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)
````

The output should be something like:

````
# Get today's date and calculate tomorrow's date
from datetime import datetime, timedelta
today = datetime.now()
tomorrow = (today + timedelta(days=1)).strftime("%Y-%m-%d")

# Define the time slots
start_time = "10:00"
end_time = "12:00"

# Check availability first
is_available = check_availability(tomorrow, start_time, end_time)

# Only proceed with making the appointment if it's available
appointment_result = (
    make_appointment(
        day=tomorrow,
        start_time=start_time,
        end_time=end_time,
        title="Meeting with Thesis Supervisor"
    )
    if is_available
    else {"appointment_made": False}
)

# Add to reminders only if the appointment was made
if appointment_result["appointment_made"]:
    add_to_reminders("Meeting with Thesis Supervisor scheduled for 10:00 AM tomorrow")
```

This code will:
1. Calculate tomorrow's date in YYYY-MM-DD format
2. Check if the 10:00-12:00 slot is available
3. If available, make the appointment with the specified details
4. If the appointment is successfully made, add a reminder to the system

The code handles all error cases implicitly through the boolean returns of the functions. If any step fails, the subsequent steps won't execute, preventing partial or invalid appointments.
````

## Evaluation & Performance

We evaluate the model on the following benchmarks:

1. Berkeley Function Calling Leaderboard (BFCL)
2. MMLU-Pro
3. **Dria-Pythonic-Agent-Benchmark (DPAB):** The benchmark we curated with a synthetic data generation +model-based validation + filtering and manual selection to evaluate LLMs on their Pythonic function calling ability, spanning multiple scenarios and tasks. More detailed information about the benchmark and the Github repo will be released soon.

Below are the BFCL results: evaluation results for ***Qwen2.5-Coder-3B-Instruct***, ***Dria-Agent-α-3B*** and ***gpt-4o-2024-11-20***

| Metric                                | Qwen/Qwen2.5-3B-Instruct   | Dria-Agent-a-3B   | gpt-4o-2024-11-20 (Prompt) |
|---------------------------------------|-----------|-----------|-----------|
| **Non-Live Simple AST**               | 75.50%    | 75.08%    | 79.42%    |
| **Non-Live Multiple AST**             | 90.00%    | 93.00%    | 95.50%    |
| **Non-Live Parallel AST**             | 80.00%    | 85.00%    | 94.00%    |
| **Non-Live Parallel Multiple AST**    | 78.50%    | 79.00%    | 83.50%    |
| **Non-Live Simple Exec**              | 82.07%    | 87.57%    | 100.00%    |
| **Non-Live Multiple Exec**            | 86.00%    | 85.14%    | 94.00%    |
| **Non-Live Parallel Exec**            | 82.00%    | 90.00%    | 86.00%    |
| **Non-Live Parallel Multiple Exec**   | 80.00%    | 88.00%    | 77.50%    |
| **Live Simple AST**                   | 68.22%    | 70.16%    | 83.72%    |
| **Live Multiple AST**                 | 66.00%    | 67.14%    | 79.77%    |
| **Live Parallel AST**                 | 62.50%    | 50.00%    | 87.50%    |
| **Live Parallel Multiple AST**        | 66.67%    | 70.83%    | 70.83%    |
| **Relevance Detection**               | 88.89%    | 100.00%   | 83.33%    |

and the MMLU-Pro and DPAB results:

| Benchmark Name | Qwen2.5-Coder-3B-Instruct | Dria-Agent-α-3B |
|----------------|---------------------------|-----------------|
| MMLU-Pro | 35.2 ([Self Reported](https://arxiv.org/pdf/2409.12186)) | 29.8* |
| DPAB (Pythonic, Strict) | 26 | 72 |

**\*Note:** The model tends to use Pythonic function calling for a lot of the test cases in STEM-related fields (math, physics, chemistry, etc.) in the MMLU-Pro benchmark, which isn't captured by the evaluation framework and scripts provided in their [Github repository](https://github.com/TIGER-AI-Lab/MMLU-Pro/tree/main). We haven't modified the script for evaluation, and leave it for the future iterations of this model. However, by performing qualitative analysis on the model responses, we suspect that the model's score will increase instead of suffering a ~6% decrease.

#### Citation

```
@misc{Dria-Agent-a,
      url={https://huggingface.co/blog/andthattoo/dria-agent-a},
      title={Dria-Agent-a},
      author={"andthattoo", "Atakan Tekparmak"}
}
```