Update README.md

README.md

@@ -1,74 +1,141 @@
 ---
-license: apache-2.0
+license: other
+license_name: qwen-research
 license_link: https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct/blob/main/LICENSE
 language:
 - en
 base_model:
-- Qwen/Qwen2.5-Coder-7B
+- Qwen/Qwen2.5-Coder-7B-Instruct
 pipeline_tag: text-generation
 library_name: transformers
 tags:
 - code
-- codeqwen
 - chat
 - qwen
 - qwen-coder
+- agent
 ---
 
-
-# Qwen2.5-Coder-7B-Instruct
+# Dria-Agent-α-7B
 
 ## Introduction
 
-Qwen2.5-Coder is the latest series of Code-Specific Qwen large language models (formerly known as CodeQwen). As of now, Qwen2.5-Coder has covered six mainstream model sizes, 0.5, 1.5, 3, 7, 14, 32 billion parameters, to meet the needs of different developers. Qwen2.5-Coder brings the following improvements upon CodeQwen1.5:
-
-- Significantly improvements in **code generation**, **code reasoning** and **code fixing**. Base on the strong Qwen2.5, we scale up the training tokens into 5.5 trillion including source code, text-code grounding, Synthetic data, etc. Qwen2.5-Coder-32B has become the current state-of-the-art open-source codeLLM, with its coding abilities matching those of GPT-4o.
-- A more comprehensive foundation for real-world applications such as **Code Agents**. Not only enhancing coding capabilities but also maintaining its strengths in mathematics and general competencies.
-- **Long-context Support** up to 128K tokens.
-
-**This repo contains the instruction-tuned 7B Qwen2.5-Coder model**, which has the following features:
-- Type: Causal Language Models
-- Training Stage: Pretraining & Post-training
-- Architecture: transformers with RoPE, SwiGLU, RMSNorm, and Attention QKV bias
-- Number of Parameters: 7.61B
-- Number of Paramaters (Non-Embedding): 6.53B
-- Number of Layers: 28
-- Number of Attention Heads (GQA): 28 for Q and 4 for KV
-- Context Length: Full 131,072 tokens
-  - Please refer to [this section](#processing-long-texts) for detailed instructions on how to deploy Qwen2.5 for handling long texts.
-  
-For more details, please refer to our [blog](https://qwenlm.github.io/blog/qwen2.5-coder-family/), [GitHub](https://github.com/QwenLM/Qwen2.5-Coder), [Documentation](https://qwen.readthedocs.io/en/latest/), [Arxiv](https://arxiv.org/abs/2409.12186).
-
-## Requirements
+***Dria-Agent-α*** are series of large language models trained on top of the [Qwen2.5-Coder](https://huggingface.co/collections/Qwen/qwen25-coder-66eaa22e6f99801bf65b0c2f) series, specifically on top of the [Qwen/Qwen2.5-Coder-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct) and [Qwen/Qwen2.5-Coder-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct) models to be used in agentic applications. These models are the first instalment of agent-focused LLMs (hence the **α** in the naming) we hope to improve with better and more elaborate techniques in subsequent releases.
 
-The code of Qwen2.5-Coder has been in the latest Hugging face `transformers` and we advise you to use the latest version of `transformers`.
+Dria-Agent-α employs ***Pythonic function calling***, which is LLMs using blocks of Python code to interact with provided tools and output actions. This method was inspired by many previous work, including but not limited to [DynaSaur](https://arxiv.org/pdf/2411.01747), [RLEF](https://arxiv.org/pdf/2410.02089), [ADAS](https://arxiv.org/pdf/2408.08435) and [CAMEL](https://arxiv.org/pdf/2303.17760). This way of function calling has a few advantages over traditional JSON-based function calling methods:
 
-With `transformers<4.37.0`, you will encounter the following error:
-```
-KeyError: 'qwen2'
-```
+1. **One-shot Parallel Multiple Function Calls:** The model can can utilise many synchronous processes in one chat turn to arrive to a solution, which would require other function calling models multiple turns of conversation.
+2. **Free-form Reasoning and Actions:** The model provides reasoning traces freely in natural language and the actions in between \`\`\`python \`\`\` blocks, as it already tends to do without special prompting or tuning. This tries to mitigate the possible performance loss caused by imposing specific formats on LLM outputs discussed in [Let Me Speak Freely?](https://arxiv.org/pdf/2408.02442)
+3. **On-the-fly Complex Solution Generation:** The solution provided by the model is essentially a Python program with the exclusion of some "risky" builtins like `exec`, `eval` and `compile` (see full list in **Quickstart** below). This enables the model to implement custom complex logic with conditionals and synchronous pipelines (using the output of one function in the next function's arguments) which would not be possible with the current JSON-based function calling methods (as far as we know).
 
 ## Quickstart
 
-Here provides a code snippet with `apply_chat_template` to show you how to load the tokenizer and model and how to generate contents.
-
-```python
+````python
+import json
+from typing import Any, Dict, List
 from transformers import AutoModelForCausalLM, AutoTokenizer
 
-model_name = "Qwen/Qwen2.5-Coder-7B-Instruct"
-
+model_name = "driaforall/Dria-Agent-a-7B"
 model = AutoModelForCausalLM.from_pretrained(
-    model_name,
-    torch_dtype="auto",
-    device_map="auto"
+    model_name, device_map="auto", torch_dtype="auto", trust_remote_code=True
 )
 tokenizer = AutoTokenizer.from_pretrained(model_name)
 
-prompt = "write a quick sort algorithm."
+# Please use our provided prompt for best performance
+SYSTEM_PROMPT = """
+You are an expert AI assistant that specializes in providing Python code to solve the task/problem at hand provided by the user.
+
+You can use Python code freely, including the following available functions:
+
+<|functions_schema|>
+{{functions_schema}}
+<|end_functions_schema|>
+
+The following dangerous builtins are restricted for security:
+- exec
+- eval
+- execfile
+- compile
+- importlib
+- input
+- exit
+
+Think step by step and provide your reasoning, outside of the function calls.
+You can write Python code and use the available functions. Provide all your python code in a SINGLE markdown code block like the following:
+
+```python
+result = example_function(arg1, "string")
+result2 = example_function2(result, arg2)
+```
+
+DO NOT use print() statements AT ALL. Avoid mutating variables whenever possible.
+""".strip()
+
+
+get_sample_data = """
+def check_availability(day: str, start_time: str, end_time: str) -> bool:
+    \"\"\" 
+    Check if a time slot is available on a given day.
+
+    Args:
+    - day: The day to check in YYYY-MM-DD format
+    - start_time: Start time in HH:MM format
+    - end_time: End time in HH:MM format
+
+    Returns:
+    - True if slot is available, False otherwise
+    \"\"\"
+    pass
+
+def make_appointment(day: str, start_time: str, end_time: str) -> dict:
+    \"\"\" 
+    Make an appointment for a given time slot.
+
+    Args:
+    - day: The day to make appointment in YYYY-MM-DD format
+    - start_time: Start time in HH:MM format
+    - end_time: End time in HH:MM format
+    - title: The title of the appointment
+
+    Returns:
+    - A dictionary with the appointment details and if it's made or not.
+        dict keys:
+            - day (str): The day the appointment is on, in YYYY-MM-DD format
+            - start_time (str): Start time in HH:MM format
+            - end_time (str): End time in HH:MM format
+            - appointment_made (bool): Whether the appointment is successfully made or not. 
+    \"\"\"
+    pass
+
+def add_to_reminders(reminder_text: str) -> bool:
+    \"\"\" 
+    Add a text to reminders.
+
+    Args: 
+    - reminder_text: The text to add to reminders
+
+    Returns:
+    - Whether the reminder was successfully created or not.
+    \"\"\"
+    pass
+"""
+
+# Helper function to create the system prompt for our model
+def format_prompt(tools: str):
+    return SYSTEM_PROMPT.format(functions_schema=tools)
+
+system_prompt = SYSTEM_PROMPT.replace("{{functions_schema}}", get_sample_data)
+
+USER_QUERY = """
+Can you check if I have tomorrow 10:00-12:00 available and make an appointment for a meeting 
+with my thesis supervisor if so? If you made the appointment, please add it to my reminders.
+"""
+
 messages = [
-    {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."},
-    {"role": "user", "content": prompt}
+    {"role": "system", "content": system_prompt},
+    {"role": "user", "content": USER_QUERY},
 ]
+
 text = tokenizer.apply_chat_template(
     messages,
     tokenize=False,
@@ -78,58 +145,86 @@ model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
 
 generated_ids = model.generate(
     **model_inputs,
-    max_new_tokens=512
+    max_new_tokens=2048
 )
 generated_ids = [
     output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
 ]
 
 response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
-```
+print(response)
+````
 
-### Processing Long Texts
-
-The current `config.json` is set for context length up to 32,768 tokens.
-To handle extensive inputs exceeding 32,768 tokens, we utilize [YaRN](https://arxiv.org/abs/2309.00071), a technique for enhancing model length extrapolation, ensuring optimal performance on lengthy texts.
-
-For supported frameworks, you could add the following to `config.json` to enable YaRN:
-```json
-{
-  ...,
-  "rope_scaling": {
-    "factor": 4.0,
-    "original_max_position_embeddings": 32768,
-    "type": "yarn"
-  }
-}
-```
+The output should be something like:
 
-For deployment, we recommend using vLLM. 
-Please refer to our [Documentation](https://qwen.readthedocs.io/en/latest/deployment/vllm.html) for usage if you are not familar with vLLM.
-Presently, vLLM only supports static YARN, which means the scaling factor remains constant regardless of input length, **potentially impacting performance on shorter texts**. 
-We advise adding the `rope_scaling` configuration only when processing long contexts is required.
+````
+To accomplish this task, we need to follow these steps:
 
-## Evaluation & Performance
+1. Check if the time slot from 10:00 to 12:00 on tomorrow is available using the `check_availability` function.
+2. If the slot is available, proceed to make an appointment using the `make_appointment` function.
+3. If the appointment is successfully made, add it to the reminders using the `add_to_reminders` function.
 
-Detailed evaluation results are reported in this [📑 blog](https://qwenlm.github.io/blog/qwen2.5-coder-family/).
+Let's break down the code into these steps:
 
-For requirements on GPU memory and the respective throughput, see results [here](https://qwen.readthedocs.io/en/latest/benchmark/speed_benchmark.html).
+```python
+from datetime import datetime, timedelta
 
-## Citation
+# Get today's date and calculate tomorrow's date
+today = datetime.now()
+tomorrow = today + timedelta(days=1)
+tomorrow_str = tomorrow.strftime('%Y-%m-%d')
 
-If you find our work helpful, feel free to give us a cite.
+# Define the time slots
+start_time = '10:00'
+end_time = '12:00'
 
+# Step 1: Check availability
+is_available = check_availability(tomorrow_str, start_time, end_time)
+
+# Step 2: Make appointment if available
+if is_available:
+    appointment_result = make_appointment(tomorrow_str, start_time, end_time, "Meeting with Thesis Supervisor")
+
+# Step 3: Add to reminders if appointment is made
+if appointment_result['appointment_made']:
+    reminder_text = f"Appointment made for {appointment_result['day']} from {appointment_result['start_time']} to {appointment_result['end_time']}."
+    add_to_reminders(reminder_text)
 ```
-@article{hui2024qwen2,
-      title={Qwen2. 5-Coder Technical Report},
-      author={Hui, Binyuan and Yang, Jian and Cui, Zeyu and Yang, Jiaxi and Liu, Dayiheng and Zhang, Lei and Liu, Tianyu and Zhang, Jiajun and Yu, Bowen and Dang, Kai and others},
-      journal={arXiv preprint arXiv:2409.12186},
-      year={2024}
-}
-@article{qwen2,
-      title={Qwen2 Technical Report}, 
-      author={An Yang and Baosong Yang and Binyuan Hui and Bo Zheng and Bowen Yu and Chang Zhou and Chengpeng Li and Chengyuan Li and Dayiheng Liu and Fei Huang and Guanting Dong and Haoran Wei and Huan Lin and Jialong Tang and Jialin Wang and Jian Yang and Jianhong Tu and Jianwei Zhang and Jianxin Ma and Jin Xu and Jingren Zhou and Jinze Bai and Jinzheng He and Junyang Lin and Kai Dang and Keming Lu and Keqin Chen and Kexin Yang and Mei Li and Mingfeng Xue and Na Ni and Pei Zhang and Peng Wang and Ru Peng and Rui Men and Ruize Gao and Runji Lin and Shijie Wang and Shuai Bai and Sinan Tan and Tianhang Zhu and Tianhao Li and Tianyu Liu and Wenbin Ge and Xiaodong Deng and Xiaohuan Zhou and Xingzhang Ren and Xinyu Zhang and Xipin Wei and Xuancheng Ren and Yang Fan and Yang Yao and Yichang Zhang and Yu Wan and Yunfei Chu and Yuqiong Liu and Zeyu Cui and Zhenru Zhang and Zhihao Fan},
-      journal={arXiv preprint arXiv:2407.10671},
-      year={2024}
-}
-```
+
+This code will first determine if the specified time slot is available tomorrow. If it is, it will attempt to make the appointment and then add it to the reminders if successful.
+````
+
+## Evaluation & Performance
+
+We evaluate the model on the following benchmarks:
+
+1. Berkeley Function Calling Leaderboard (BFCL)
+2. MMLU-Pro
+3. **Dria-Pythonic-Agent-Benchmark (DPAB):** The benchmark we curated with a synthetic data generation +model-based validation + filtering and manual selection to evaluate LLMs on their Pythonic function calling ability, spanning multiple scenarios and tasks. More detailed information about the benchmark and the Github repo will be released soon.
+
+Below are the BFCL results: evaluation results for ***Qwen2.5-Coder-3B-Instruct***, ***Dria-Agent-α-3B*** and ***gpt-4o-2024-11-20***
+
+| Metric                                | Qwen/Qwen2.5-3B-Instruct   | Dria-Agent-a-3B   | Dria-Agent-a-7B   | gpt-4o-2024-11-20 (Prompt)   |
+|---------------------------------------|-----------|-----------|-----------|-----------|
+| **Non-Live Simple AST**               | 75.50%    | 75.08%    | 77.83%    | 79.42%    |
+| **Non-Live Multiple AST**             | 90.00%    | 93.00%    | 94.50%    | 95.50%    |
+| **Non-Live Parallel AST**             | 80.00%    | 85.00%    | 87.00%    | 94.00%    |
+| **Non-Live Parallel Multiple AST**    | 78.50%    | 79.00%    | 88.00%    | 83.50%    |
+| **Non-Live Simple Exec**              | 82.07%    | 87.57%    | 80.00%    | 100.00%   |
+| **Non-Live Multiple Exec**            | 86.00%    | 85.14%    | 84.00%    | 94.00%    |
+| **Non-Live Parallel Exec**            | 82.00%    | 90.00%    | 70.00%    | 86.00%    |
+| **Non-Live Parallel Multiple Exec**   | 80.00%    | 88.00%    | 65.00%    | 77.50%    |
+| **Live Simple AST**                   | 68.22%    | 70.16%    | 82.95%    | 83.72%    |
+| **Live Multiple AST**                 | 66.00%    | 67.14%    | 78.25%    | 79.77%    |
+| **Live Parallel AST**                 | 62.50%    | 50.00%    | 81.25%    | 87.50%    |
+| **Live Parallel Multiple AST**        | 66.67%    | 70.83%    | 70.83%    | 70.83%    |
+| **Relevance Detection**               | 88.89%    | 100.00%   | 100.00%   | 83.33%    |
+
+and the MMLU-Pro and DPAB results:
+
+| Benchmark Name | Qwen2.5-Coder-7B-Instruct | Dria-Agent-α-7B |
+|----------------|---------------------------|-----------------|
+| MMLU-Pro | 45.6 ([Self Reported](https://arxiv.org/pdf/2409.12186)) | TBD |
+| DPAB (Pythonic, Strict) | 24 | 51 |
+
+**\*Note:** The model tends to use Pythonic function calling for a lot of the test cases in STEM-related fields (math, physics, chemistry, etc.) in the MMLU-Pro benchmark, which isn't captured by the evaluation framework and scripts provided in their [Github repository](https://github.com/TIGER-AI-Lab/MMLU-Pro/tree/main). We haven't modified the script for evaluation, and leave it for the future iterations of this model. However, by performing qualitative analysis on the model responses, we suspect that the model's score will increase instead of suffering a ~6% decrease.