
(Works with [Mobile-Env v3.x](https://github.com/X-LANCE/Mobile-Env/tree/v3.0).)

# WikiHow Task Set

WikiHow task set is an InfoUI interaction task set based on
[Mobile-Env](https://github.com/X-LANCE/Mobile-Env) proposed in [*Mobile-Env:
An Evaluation Platform and Benchmark for Interactive Agents in LLM
Era*](https://arxiv.org/abs/2305.08144).
[WikiHow](https://www.wikihow.com/Main-Page) is a collaborative wiki site about
various real-life tips with more than 340,000 online articles. To construct the
task set, 107,448 pages are crawled, and the dumped website data occupy about
88 GiB totally.

Several task definition templates are designed according to the functions of
WikiHow app and task definitions are instantiated through the template toolkit
in Mobile-Env. 577 tasks are sampled from the extended set, which is named the
*canonical set* (`wikihow-canonical.tar.xz`). Owing to the limit of the
budgets, only 150 tasks are tested using the proposed LLM-based agent.  These
150 tasks are given in `wikihow-microcanon.tar.xz`.  We call it the *canonical
subset* or the *micro canonical set*.

### Website Data Replay

The replay script for [mitmproxy](https://mitmproxy.org/) is given as
`replay_url.py`. To use this replay script, the information retrieval tool
[Pyserini](https://github.com/castorini/pyserini/) is required. Four parameters
are expected to be assigned in the script:

+ The crawled data from WikiHow website (`dumps` in `wikihow.data.tar.xz`)
+ The HTML templates used to mock the search result page (`templates` in
  `wikihow.data.tar.xz`)
+ The indices for the search engine based on Pyserini (`indices-t/indices` in
  `wikihow.data.tar.xz`)
+ The metadata of the crawled articles (`indices-t/docs/doc_meta.csv` in
  `wikihow.data.tar.xz`)

All the required data are offered in `wikihow.data.tar.xz`. (The archive is
about 78 GiB. And the decompressed data are about 88 GiB.) The archive is split
into two pieces (`wikihow.data.tar.xz.00` and `wikihow.data.tar.xz.01`). You
can use `cat` to concatenate them:

```sh
cat wikihow.data.tar.xz.00 wikihow.data.tar.xz.01 >wikihow.data.tar.xz
```

The SHA256 checksums are provided in `wikihow.data.tar.xz.sha256` to check the
integrity.

To run the script:

```sh
mitmproxy --showhost -s replay_url.py
```

### Certificate Unpinning Plan

The `syscert` plan proposed by Mobile-Env works for WikiHow app. You can
complete the config according to the [guideline of
Mobile-Env](https://github.com/X-LANCE/Mobile-Env/blob/master/docs/dynamic-app-en.md).
The available APK package from [APKCombo](https://apkcombo.com/) is provided.
And note to use the AVD image of version Android 11.0 (API Level 30) (Google
APIs) to obtain the best compatibility and the root-enabled ADBD.

### Human-Rewritten Instructions

Human-rewritten instructions for the *canonical set* are release under
`instruction_rewriting/`. An AndroidEnv wrapper `InstructionRewritingWrapper`
is provided to load the rewritten instructions (`merged_doccano.json`) and
public patterns (`pattern-*.txt`). The annotations are collected via
[doccano](https://doccano.github.io/doccano/). The patterns are parsed by
[`sentence_pattern.py`](instruction_rewriting/sentence_pattern.py).

license:apache-2.0 arxiv:2305.08144 region:us
{"license": "apache-2.0"}
TAGS
#license-apache-2.0 #arxiv-2305.08144 #region-us 


(Works with Mobile-Env v3.x.)

# WikiHow Task Set

WikiHow task set is an InfoUI interaction task set based on
Mobile-Env proposed in *Mobile-Env:
An Evaluation Platform and Benchmark for Interactive Agents in LLM
Era*.
WikiHow is a collaborative wiki site about
various real-life tips with more than 340,000 online articles. To construct the
task set, 107,448 pages are crawled, and the dumped website data occupy about
88 GiB totally.

Several task definition templates are designed according to the functions of
WikiHow app and task definitions are instantiated through the template toolkit
in Mobile-Env. 577 tasks are sampled from the extended set, which is named the
*canonical set* ('URL'). Owing to the limit of the
budgets, only 150 tasks are tested using the proposed LLM-based agent.  These
150 tasks are given in 'URL'.  We call it the *canonical
subset* or the *micro canonical set*.

### Website Data Replay

The replay script for mitmproxy is given as
'replay_url.py'. To use this replay script, the information retrieval tool
Pyserini is required. Four parameters
are expected to be assigned in the script:

+ The crawled data from WikiHow website ('dumps' in 'URL')
+ The HTML templates used to mock the search result page ('templates' in
  'URL')
+ The indices for the search engine based on Pyserini ('indices-t/indices' in
  'URL')
+ The metadata of the crawled articles ('indices-t/docs/doc_meta.csv' in
  'URL')

All the required data are offered in 'URL'. (The archive is
about 78 GiB. And the decompressed data are about 88 GiB.) The archive is split
into two pieces ('URL.00' and 'URL.01'). You
can use 'cat' to concatenate them:



The SHA256 checksums are provided in 'URL.sha256' to check the
integrity.

To run the script:



### Certificate Unpinning Plan

The 'syscert' plan proposed by Mobile-Env works for WikiHow app. You can
complete the config according to the guideline of
Mobile-Env.
The available APK package from APKCombo is provided.
And note to use the AVD image of version Android 11.0 (API Level 30) (Google
APIs) to obtain the best compatibility and the root-enabled ADBD.

### Human-Rewritten Instructions

Human-rewritten instructions for the *canonical set* are release under
'instruction_rewriting/'. An AndroidEnv wrapper 'InstructionRewritingWrapper'
is provided to load the rewritten instructions ('merged_doccano.json') and
public patterns ('pattern-*.txt'). The annotations are collected via
doccano. The patterns are parsed by
'sentence_pattern.py'.

# WikiHow Task Set

WikiHow task set is an InfoUI interaction task set based on
Mobile-Env proposed in *Mobile-Env:
An Evaluation Platform and Benchmark for Interactive Agents in LLM
Era*.
WikiHow is a collaborative wiki site about
various real-life tips with more than 340,000 online articles. To construct the
task set, 107,448 pages are crawled, and the dumped website data occupy about
88 GiB totally.

Several task definition templates are designed according to the functions of
WikiHow app and task definitions are instantiated through the template toolkit
in Mobile-Env. 577 tasks are sampled from the extended set, which is named the
*canonical set* ('URL'). Owing to the limit of the
budgets, only 150 tasks are tested using the proposed LLM-based agent.  These
150 tasks are given in 'URL'.  We call it the *canonical
subset* or the *micro canonical set*. ### Website Data Replay

The replay script for mitmproxy is given as
'replay_url.py'. To use this replay script, the information retrieval tool
Pyserini is required. Four parameters
are expected to be assigned in the script:

+ The crawled data from WikiHow website ('dumps' in 'URL')
+ The HTML templates used to mock the search result page ('templates' in
  'URL')
+ The indices for the search engine based on Pyserini ('indices-t/indices' in
  'URL')
+ The metadata of the crawled articles ('indices-t/docs/doc_meta.csv' in
  'URL')

All the required data are offered in 'URL'. (The archive is
about 78 GiB. And the decompressed data are about 88 GiB.) The archive is split
into two pieces ('URL.00' and 'URL.01'). You
can use 'cat' to concatenate them:



The SHA256 checksums are provided in 'URL.sha256' to check the
integrity.

To run the script: ### Certificate Unpinning Plan

The 'syscert' plan proposed by Mobile-Env works for WikiHow app. You can
complete the config according to the guideline of
Mobile-Env.
The available APK package from APKCombo is provided.
And note to use the AVD image of version Android 11.0 (API Level 30) (Google
APIs) to obtain the best compatibility and the root-enabled ADBD. ### Human-Rewritten Instructions

Human-rewritten instructions for the *canonical set* are release under
'instruction_rewriting/'. An AndroidEnv wrapper 'InstructionRewritingWrapper'
is provided to load the rewritten instructions ('merged_doccano.json') and
public patterns ('pattern-*.txt'). The annotations are collected via
doccano. The patterns are parsed by
'sentence_pattern.py'.
TAGS
#license-apache-2.0 #arxiv-2305.08144 #region-us 
 # WikiHow Task Set

WikiHow task set is an InfoUI interaction task set based on
Mobile-Env proposed in *Mobile-Env:
An Evaluation Platform and Benchmark for Interactive Agents in LLM
Era*.
WikiHow is a collaborative wiki site about
various real-life tips with more than 340,000 online articles. To construct the
task set, 107,448 pages are crawled, and the dumped website data occupy about
88 GiB totally.

Several task definition templates are designed according to the functions of
WikiHow app and task definitions are instantiated through the template toolkit
in Mobile-Env. 577 tasks are sampled from the extended set, which is named the
*canonical set* ('URL'). Owing to the limit of the
budgets, only 150 tasks are tested using the proposed LLM-based agent.  These
150 tasks are given in 'URL'.  We call it the *canonical
subset* or the *micro canonical set*. ### Website Data Replay

The replay script for mitmproxy is given as
'replay_url.py'. To use this replay script, the information retrieval tool
Pyserini is required. Four parameters
are expected to be assigned in the script:

+ The crawled data from WikiHow website ('dumps' in 'URL')
+ The HTML templates used to mock the search result page ('templates' in
  'URL')
+ The indices for the search engine based on Pyserini ('indices-t/indices' in
  'URL')
+ The metadata of the crawled articles ('indices-t/docs/doc_meta.csv' in
  'URL')

All the required data are offered in 'URL'. (The archive is
about 78 GiB. And the decompressed data are about 88 GiB.) The archive is split
into two pieces ('URL.00' and 'URL.01'). You
can use 'cat' to concatenate them:



The SHA256 checksums are provided in 'URL.sha256' to check the
integrity.

To run the script: ### Certificate Unpinning Plan

The 'syscert' plan proposed by Mobile-Env works for WikiHow app. You can
complete the config according to the guideline of
Mobile-Env.
The available APK package from APKCombo is provided.
And note to use the AVD image of version Android 11.0 (API Level 30) (Google
APIs) to obtain the best compatibility and the root-enabled ADBD. ### Human-Rewritten Instructions

Human-rewritten instructions for the *canonical set* are release under
'instruction_rewriting/'. An AndroidEnv wrapper 'InstructionRewritingWrapper'
is provided to load the rewritten instructions ('merged_doccano.json') and
public patterns ('pattern-*.txt'). The annotations are collected via
doccano. The patterns are parsed by
'sentence_pattern.py'.
passage: TAGS
#license-apache-2.0 #arxiv-2305.08144 #region-us 
# WikiHow Task Set

WikiHow task set is an InfoUI interaction task set based on
Mobile-Env proposed in *Mobile-Env:
An Evaluation Platform and Benchmark for Interactive Agents in LLM
Era*.
WikiHow is a collaborative wiki site about
various real-life tips with more than 340,000 online articles. To construct the
task set, 107,448 pages are crawled, and the dumped website data occupy about
88 GiB totally.

Several task definition templates are designed according to the functions of
WikiHow app and task definitions are instantiated through the template toolkit
in Mobile-Env. 577 tasks are sampled from the extended set, which is named the
*canonical set* ('URL'). Owing to the limit of the
budgets, only 150 tasks are tested using the proposed LLM-based agent.  These
150 tasks are given in 'URL'.  We call it the *canonical
subset* or the *micro canonical set*.### Website Data Replay

The replay script for mitmproxy is given as
'replay_url.py'. To use this replay script, the information retrieval tool
Pyserini is required. Four parameters
are expected to be assigned in the script:

+ The crawled data from WikiHow website ('dumps' in 'URL')
+ The HTML templates used to mock the search result page ('templates' in
  'URL')
+ The indices for the search engine based on Pyserini ('indices-t/indices' in
  'URL')
+ The metadata of the crawled articles ('indices-t/docs/doc_meta.csv' in
  'URL')

All the required data are offered in 'URL'. (The archive is
about 78 GiB. And the decompressed data are about 88 GiB.) The archive is split
into two pieces ('URL.00' and 'URL.01'). You
can use 'cat' to concatenate them:



The SHA256 checksums are provided in 'URL.sha256' to check the
integrity.

To run the script: