finish

README.md

@@ -43,124 +43,36 @@ Please refer to our [GitHub](https://github.com/THUDM/CogAgent) for specific exa
 conversations but does support continuous execution history. Below are guidelines on how users should format their input
 for the model and interpret the formatted output.
 
-### User Input
-
-1. **`task` field**  
-   A task description provided by the user, similar to a textual prompt. The input should be concise and clear to guide
-   the `CogAgent-9B-20241220` model to complete the task.
-
-2. **`platform` field**  
-   `CogAgent-9B-20241220` supports operation on several platforms with GUI interfaces:
-    - **Windows**: Use the `WIN` field for Windows 10 or 11.
-    - **Mac**: Use the `MAC` field for Mac 14 or 15.
-    - **Mobile**: Use the `Mobile` field for Android 13, 14, 15, or similar Android-based UI versions.
-
-   If using other systems, results may vary. Use the `Mobile` field for mobile devices, `WIN` for Windows, and `MAC` for
-   Mac.
-
-3. **`format` field**  
-   Specifies the desired format of the returned data. Options include:
-    - `Answer in Action-Operation-Sensitive format.`: The default format in our demo, returning actions, corresponding
-      operations, and sensitivity levels.
-    - `Answer in Status-Plan-Action-Operation format.`: Returns status, plans, actions, and corresponding operations.
-    - `Answer in Status-Action-Operation-Sensitive format.`: Returns status, actions, corresponding operations, and
-      sensitivity levels.
-    - `Answer in Status-Action-Operation format.`: Returns status, actions, and corresponding operations.
-    - `Answer in Action-Operation format.`: Returns actions and corresponding operations.
-
-4. **`history` field**  
-   The input should be concatenated in the following order:
-   ```
-   query = f'{task}{history}{platform}{format}'
-   ```
-
-### Model Output
-
-1. **Sensitive Operations**: Includes types like `<<Sensitive Operation>>` or `<<General Operation>>`, returned only
-   when `Sensitive` is requested.
-2. **`Plan`, `Agent`, `Status`, `Action` fields**: Describe the model's behavior and operations, returned based on the
-   requested format.
-3. **General Responses**: Summarizes the output before formatting.
-4. **`Grounded Operation` field**: Describes the model's specific actions, such as coordinates, element types, and
-   descriptions. Actions include:
-    - `CLICK`: Simulates mouse clicks or touch gestures.
-    - `LONGPRESS`: Simulates long presses (supported only in `Mobile` mode).
-
-### Example
-
-If the user wants to mark all emails as read on a Mac system and requests an `Action-Operation-Sensitive` format, the
-prompt should be:
+## Run the Model
 
-```
-Task: Mark all emails as read
-(Platform: Mac)
-(Answer in Action-Operation-Sensitive format.)
-```
-
-Below are examples of model responses based on different requested formats:
-
-<details>
-<summary>Answer in Action-Operation-Sensitive format</summary>
-
-```
-Action: Click the "Mark All as Read" button at the top toolbar to mark all emails as read.
-Grounded Operation: CLICK(box=[[219,186,311,207]], element_type='Clickable Text', element_info='Mark All as Read')
-<<General Operation>>
-```
-
-</details>
-
-<details>
-<summary>Answer in Status-Plan-Action-Operation format</summary>
+<p>Please visit our <a href="https://github.com/THUDM/CogAgent">github</a> for specific running examples, as well as the part for prompt concatenation <strong style="color: red;">(this directly affects whether the model runs correctly)</strong>.</p>
 
-```
-Status: None
-Plan: None
-Action: Click the "Mark All as Read" button at the top center of the inbox page to mark all emails as read.
-Grounded Operation: CLICK(box=[[219,186,311,207]], element_type='Clickable Text', element_info='Mark All as Read')
-```
+In particular, pay attention to the prompt concatenation process.
+You can refer to [app/client.py#L115](https://github.com/THUDM/CogAgent/blob/e3ca6f4dc94118d3dfb749f195cbb800ee4543ce/app/client.py#L115) for concatenating user input prompts.
+``` python
+current_platform = identify_os() # "Mac" or "WIN" or "Mobile"，注意大小写
+platform_str = f"(Platform: {current_platform})\n"
+format_str = "(Answer in Action-Operation-Sensitive format.)\n" # You can use other format to replace "Action-Operation-Sensitive"
 
-</details>
+history_str = "\nHistory steps: "
+for index, (grounded_op_func, action) in enumerate(zip(history_grounded_op_funcs, history_actions)):
+   history_str += f"\n{index}. {grounded_op_func}\t{action}" # start from 0. 
 
-<details>
-<summary>Answer in Status-Action-Operation-Sensitive format</summary>
+query = f"Task: {task}{history_str}\n{platform_str}{format_str}" # Be careful about the \n
 
 ```
-Status: Currently on the email interface [[0, 2, 998, 905]], with email categories on the left [[1, 216, 144, 570]] and the inbox in the center [[144, 216, 998, 903]]. The "Mark All as Read" button [[223, 178, 311, 210]] has been clicked.
-Action: Click the "Mark All as Read" button at the top toolbar to mark all emails as read.
-Grounded Operation: CLICK(box=[[219,186,311,207]], element_type='Clickable Text', element_info='Mark All as Read')
-<<General Operation>>
-```
-
-</details>
-
-<details>
-<summary>Answer in Status-Action-Operation format</summary>
 
+A minimal user input concatenation code is as follows:
 ```
-Status: None
-Action: On the inbox page, click the "Mark All as Read" button to mark all emails as read.
-Grounded Operation: CLICK(box=[[219,186,311,207]], element_type='Clickable Text', element_info='Mark All as Read')
+"Task: Search for doors, click doors on sale and filter by brands \"Mastercraft\".\nHistory steps: \n0. CLICK(box=[[352,102,786,139]], element_info='Search')\tLeft click on the search box located in the middle top of the screen next to the Menards logo.\n1. TYPE(box=[[352,102,786,139]], text='doors', element_info='Search')\tIn the search input box at the top, type 'doors'.\n2. CLICK(box=[[787,102,809,139]], element_info='SEARCH')\tLeft click on the magnifying glass icon next to the search bar to perform the search.\n3. SCROLL_DOWN(box=[[0,209,998,952]], step_count=5, element_info='[None]')\tScroll down the page to see the available doors.\n4. CLICK(box=[[280,708,710,809]], element_info='Doors on Sale')\tClick the \"Doors On Sale\" button in the middle of the page to view the doors that are currently on sale.\n(Platform: WIN)\n(Answer in Action-Operation format.)\n"
 ```
-
-</details>
-
-<details>
-<summary>Answer in Action-Operation format</summary>
+The concatenated Python string will look like:
 
 ```
-Action: Right-click on the first email in the left-side list to open the action menu.
-Grounded Operation: RIGHT_CLICK(box=[[154,275,343,341]], element_info='[AXCell]')
+"Task: Search for doors, click doors on sale and filter by brands \"Mastercraft\".\nHistory steps: \n0. CLICK(box=[[352,102,786,139]], element_info='Search')\tLeft click on the search box located in the middle top of the screen next to the Menards logo.\n1. TYPE(box=[[352,102,786,139]], text='doors', element_info='Search')\tIn the search input box at the top, type 'doors'.\n2. CLICK(box=[[787,102,809,139]], element_info='SEARCH')\tLeft click on the magnifying glass icon next to the search bar to perform the search.\n3. SCROLL_DOWN(box=[[0,209,998,952]], step_count=5, element_info='[None]')\tScroll down the page to see the available doors.\n4. CLICK(box=[[280,708,710,809]], element_info='Doors on Sale')\tClick the \"Doors On Sale\" button in the middle of the page to view the doors that are currently on sale.\n(Platform: WIN)\n(Answer in Action-Operation format.)\n"
 ```
 
-</details>
-
-### Notes
-
-1. This model is not a conversational model and does not support continuous dialogue. Please send specific instructions
-   and use the suggested concatenation method.
-2. Images must be provided as input; textual prompts alone cannot execute GUI agent tasks.
-3. The model outputs strictly formatted STR data and does not support JSON format.
+Due to the length, if you would like to understand the meaning and representation of each field in detail, please refer to the [GitHub](https://github.com/THUDM/CogAgent).
 
 ## Previous Work
 

README_zh.md

@@ -14,137 +14,42 @@
 双语开源VLM基座模型，通过数据的采集与优化、多阶段训练与策略改进等方法，`CogAgent-9B-20241220` 在GUI
 感知、推理预测准确性、动作空间完善性、任务的普适和泛化性上得到了大幅提升，能够接受中英文双语的屏幕截图和语言交互。
 此版CogAgent模型已被应用于智谱AI的 [GLM-PC产品](https://cogagent.aminer.cn/home)
-。我们希望这版模型的发布能够帮助到学术研究者们和开发者们，一起推进基于视觉语言模型的 GUI agent 的研究和应用。
+。我们希望这版模型的发布能够帮助到学术研究者们和开发者们，一起推进基于视觉语言往我们的模型的 GUI agent 的研究和应用。
 
 ## 运行模型
 
-请前往我们的[github](https://github.com/THUDM/CogAgent) 查看具体的运行示例。
+<p>请前往我们的 <a href="https://github.com/THUDM/CogAgent">github</a> 查看具体的运行示例，以及模型提示词拼接部分 <strong style="color: red;">(这直接影响模型是否正常运行)</strong>。</p>
 
-## 输入和输出
+其中，特别注意提示词拼接过程。
+您可以参考 [app/client.py#L115](https://github.com/THUDM/CogAgent/blob/e3ca6f4dc94118d3dfb749f195cbb800ee4543ce/app/client.py#L115)
+拼接用户输入提示词。
+``` python
 
-cogagent-9b-20241220是一个Agent类执行模型而非对话模型，不支持连续对话，但是但支持连续的执行历史。
-这里展示了用户应该怎么整理自己的输入格式化的传入给模型。并获得模型规则的回复。
+current_platform = identify_os() # "Mac" or "WIN" or "Mobile"，注意大小写
+platform_str = f"(Platform: {current_platform})\n"
+format_str = "(Answer in Action-Operation-Sensitive format.)\n" # You can use other format to replace "Action-Operation-Sensitive"
 
-### 用户输入部分
+history_str = "\nHistory steps: "
+for index, (grounded_op_func, action) in enumerate(zip(history_grounded_op_funcs, history_actions)):
+   history_str += f"\n{index}. {grounded_op_func}\t{action}" # start from 0. 
 
-1. `task` 字段
-
-   用户输入的任务描述，类似文本格式的prompt，该输入可以指导 CogAgent1.5 模型完成用户任务指令。请保证简洁明了。
-
-2. `platform` 字段
-
-   CogAgent1.5 支持在多个平台上执行可操作Agent功能, 我们支持的带有图形界面的操作系统有三个系统，
-    - Windows 10，11，请使用 `WIN` 字段。
-    - Mac 14，15，请使用 `MAC` 字段。
-    - Android 13，14，15 以及其他GUI和UI操作方式几乎相同的安卓UI发行版，请使用 `Mobile` 字段。
-
-   如果您使用的是其他系统，效果可能不佳，但可以尝试使用 `Mobile` 字段用于手机设备，`WIN` 字段用于Windows设备，`MAC`
-   字段用于Mac设备。
-
-3. `format` 字段
-
-   用户希望 CogAgent1.5 返回何种格式的数据, 这里有以下几种选项:
-    - `Answer in Action-Operation-Sensitive format.`: 本仓库中demo默认使用的返回方式，返回模型的行为，对应的操作，以及对应的敏感程度。
-    - `Answer in Status-Plan-Action-Operation format.`: 返回模型的装题，行为，以及相应的操作。
-    - `Answer in Status-Action-Operation-Sensitive format.`: 返回模型的状态，行为，对应的操作，以及对应的敏感程度。
-    - `Answer in Status-Action-Operation format.`: 返回模型的状态，行为。
-    - `Answer in Action-Operation format.` 返回模型的行为，对应的操作。
-
-4. `history` 字段
-
-   拼接顺序和结果应该如下所示：
-   ```
-   query = f'{task}{history}{platform}{format}'
-   ```
-
-### 模型返回部分
-
-1. 敏感操作: 包括 `<<敏感操作>> <<一般操作>>` 几种类型，只有要求返回`Sensitive`的时候返回。
-2. `Plan`, `Agent`, `Status`, `Action` 字段: 用于描述模型的行为和操作。只有要求返回对应字段的时候返回，例如带有`Action`则返回
-   `Action`字段内容。
-3. 常规回答部分，这部分回答会在格式化回答之前，表示综述。
-4. `Grounded Operation` 字段:
-   用于描述模型的具体操作，包括操作的位置，类型，以及具体的操作内容。其中 `box` 代表执行区域的坐标，`element_type` 代表执行的元素类型，
-   `element_info` 代表执行的元素描述。这些信息被一个 `操作指令` 操作所包裹。这些指令包括：
-    - `CLICK`: 点击操作，模拟鼠标点击或者手指触摸。
-    - `LONGPRESS`: 长案操作。仅在 `Mobile` 模式下支持。
-
-### 例子
-
-用户的任务是希望帮忙将所有邮件标记为已读，用户使用的是 Mac系统，希望返回的是Action-Operation-Sensitive格式。
-正确拼接后的提示词应该为：
+query = f"Task: {task}{history_str}\n{platform_str}{format_str}" # Be careful about the \n
 
 ```
-Task: 帮我将所有的邮件标注为已读
-(Platform: Mac)
-(Answer in Action-Operation-Sensitive format.)
-```
-
-接着，这里展现了不同格式要求下的返���结果:
 
-
-<details>
-<summary>Answer in Action-Operation-Sensitive format</summary>
+一个最简用户输入拼接代码如下所示：
 
 ```
-Action: 点击页面顶部工具栏中的“全部标为已读”按钮，将所有邮件标记为已读。
-Grounded Operation: CLICK(box=[[219,186,311,207]], element_type='可点击文本', element_info='全部标为已读')
-<<一般操作>>
+"Task: Search for doors, click doors on sale and filter by brands \"Mastercraft\".\nHistory steps: \n0. CLICK(box=[[352,102,786,139]], element_info='Search')\tLeft click on the search box located in the middle top of the screen next to the Menards logo.\n1. TYPE(box=[[352,102,786,139]], text='doors', element_info='Search')\tIn the search input box at the top, type 'doors'.\n2. CLICK(box=[[787,102,809,139]], element_info='SEARCH')\tLeft click on the magnifying glass icon next to the search bar to perform the search.\n3. SCROLL_DOWN(box=[[0,209,998,952]], step_count=5, element_info='[None]')\tScroll down the page to see the available doors.\n4. CLICK(box=[[280,708,710,809]], element_info='Doors on Sale')\tClick the \"Doors On Sale\" button in the middle of the page to view the doors that are currently on sale.\n(Platform: WIN)\n(Answer in Action-Operation format.)\n"
 ```
 
-</details>
-
-<details>
-<summary>Answer in Status-Plan-Action-Operation format</summary>
+拼接后的python字符串形如：
 
+``` python
+"Task: Search for doors, click doors on sale and filter by brands \"Mastercraft\".\nHistory steps: \n0. CLICK(box=[[352,102,786,139]], element_info='Search')\tLeft click on the search box located in the middle top of the screen next to the Menards logo.\n1. TYPE(box=[[352,102,786,139]], text='doors', element_info='Search')\tIn the search input box at the top, type 'doors'.\n2. CLICK(box=[[787,102,809,139]], element_info='SEARCH')\tLeft click on the magnifying glass icon next to the search bar to perform the search.\n3. SCROLL_DOWN(box=[[0,209,998,952]], step_count=5, element_info='[None]')\tScroll down the page to see the available doors.\n4. CLICK(box=[[280,708,710,809]], element_info='Doors on Sale')\tClick the \"Doors On Sale\" button in the middle of the page to view the doors that are currently on sale.\n(Platform: WIN)\n(Answer in Action-Operation format.)\n"
 ```
-Status: None
-Plan: None.
-Action: 点击收件箱页面顶部中间的“全部标记为已读”按钮，将所有邮件标记为已读。
-Grounded Operation: CLICK(box=[[219,186,311,207]], element_type='可点击文本', element_info='全部标为已读')
-```
-
-</details>
-
-<details>
-<summary>Answer in Status-Action-Operation-Sensitive format</summary>
-
-```
-Status: 当前处于邮箱界面[[0, 2, 998, 905]]，左侧是邮箱分类[[1, 216, 144, 570]]，中间是收件箱[[144, 216, 998, 903]]，已经点击“全部标为已读”按钮[[223, 178, 311, 210]]。
-Action: 点击页面顶部工具栏中的“全部标为已读”按钮，将所有邮件标记为已读。
-Grounded Operation: CLICK(box=[[219,186,311,207]], element_type='可点击文本', element_info='全部标为已读')
-<<一般操作>>
-```
-
-</details>
-
-<details>
-<summary>Answer in Status-Action-Operation format</summary>
-
-```
-Status: None
-Action: 在收件箱页面顶部，点击“全部标记为已读”按钮，将所有邮件标记为已读。
-Grounded Operation: CLICK(box=[[219,186,311,207]], element_type='可点击文本', element_info='全部标为已读')
-```
-
-</details>
-
-<details>
-<summary>Answer in Action-Operation format</summary>
-
-```
-Action: 在左侧邮件列表中，右键单击第一封邮件，以打开操作菜单。
-Grounded Operation: RIGHT_CLICK(box=[[154,275,343,341]], element_info='[AXCell]')
-```
-
-</details>
-
-### 注意事项
-
-1. 该模型不是对话模型，不支持连续对话，请发送具体指令，并参考我们提供的历史拼接方式进行拼接。
-2. 该模型必须要有图片传入，纯文字对话无法实现GUI Agent任务。
-3. 该模型输出有严格的格式要求，请严格按照我们的要求进行解析。输出格式为 STR 格式，不支持输出JSON 格式。
 
+由于篇幅较长，若您想仔细了解每个字段的含义和表示，请参考[github](https://github.com/THUDM/CogAgent)。
 
 ## 先前的工作