<!-- To update the above `dataset_info` section, please run the following command: `datasets-cli test open_australian_legal_corpus.py --save_info --all_configs`. -->

# **Open Australian Legal Corpus ‍⚖️**
<a href="https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus" alt="Release"><img src="https://img.shields.io/badge/release-v4.2.1-green"></a>

The Open Australian Legal Corpus is the first and only multijurisdictional open corpus of Australian legislative and judicial documents.

Comprised of 202,260 texts totalling over 50 million lines and 1.2 billion tokens, the Corpus includes every in force statute and regulation in the Commonwealth, New South Wales, Queensland, Western Australia, South Australia, Tasmania and Norfolk Island, in addition to thousands of bills and hundreds of thousands of court and tribunal decisions.

As the largest free and open database of its kind to date, the Corpus is intended to progress the burgeoning field of legal AI research in Australia by allowing researchers to pretrain and finetune machine learning models for downstream natural language processing tasks applied to the Australian legal domain such as document classification, summarisation, information retrieval and question answering.

To ensure its accessibility to as wide an audience as possible, the Corpus and all its documents are distributed under open source licences that, in most cases, allow for both non-commercial and commercial usage (see the [Licence 📄](https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus/blob/main/LICENCE.md)).

Those interested in learning more about the Corpus are encouraged to read Umar Butler's accompanying article, [*How I built the largest open database of Australian law*](https://umarbutler.com/how-i-built-the-largest-open-database-of-australian-law/).

## Usage 👩‍💻
The below code snippet illustrates how the Corpus may be streamed or loaded with the [Hugging Face Datasets](https://huggingface.co/docs/datasets/index) Python library:
```python
from datasets import load_dataset

corpus = load_dataset('umarbutler/open_australian_legal_corpus', split='train', streaming=True) # Set `streaming` to `False` if you wish to load the entire corpus into memory.
```

To speed up the loading of the Corpus, you may wish to install [`orjson`](https://github.com/ijl/orjson).

## Statistics 📊
The Corpus is comprised of 202,260 documents, totalling 52,613,824 lines and 1,290,869,784 tokens.

A breakdown of the number of documents by type and source is provided below:
| Source                          |   Primary Legislation |   Secondary Legislation |   Bills |   Decisions |   **Total** |
|:--------------------------------|----------------------:|------------------------:|--------:|------------:|--------:|
| Federal Register of Legislation |                  3,872 |                   19,587 |       0 |           0 |**23,459**|
| Federal Court of Australia      |                     0 |                       0 |       0 |       46,733 |**46,733**|
| High Court of Australia         |                     0 |                       0 |       0 |        9,433 |**9,433**|
| NSW Caselaw                     |                     0 |                       0 |       0 |      111,882 |**111,882**|
| NSW Legislation                 |                  1,428 |                     800 |       0 |           0 |**2,228**|
| Queensland Legislation          |                   564 |                     426 |    2,247 |           0 |**3,237**|
| Western Australian Legislation  |                   812 |                     760 |       0 |           0 |**1,572**|
| South Australian Legislation    |                   557 |                     471 |     154 |           0 |**1,182**|
| Tasmanian Legislation           |                   858 |                    1,676 |       0 |           0 |**2,534**|
| **Total**                           |**8,091**|**23,720**|**2,401**|**168,048**|**202,260**|

## Structure 🗂️
The Corpus is stored in [corpus.jsonl](https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus/blob/main/corpus.jsonl), a json lines file where each line represents a document consisting of six keys:
| Key | Description |
| --- | --- |
| version_id | A unique identifier for the current version of the document. |
| type | The type of the document. Possible values are `primary_legislation`, `secondary_legislation`, `bill` and `decision`. |
| jurisdiction | The jurisdiction of the document. Possible values are `commonwealth`, `new_south_wales`, `queensland`, `western_australia`, `south_australia`, `tasmania` and `norfolk_island`. |
| source | The source of the document. Possible values are `federal_register_of_legislation`, `federal_court_of_australia`, `high_court_of_australia`, `nsw_caselaw`, `nsw_legislation`, `queensland_legislation`, `western_australian_legislation`, `south_australian_legislation` and `tasmanian_legislation`. |
| citation | The title of the document with, in the case of legislation and bills, an abbreviated form of the document's jurisdiction enclosed in parentheses appended. |
| url | A hyperlink to the document. |
| text | The UTF-8 encoded text of the document. |

## Collection 📥
Documents were sourced from the [Federal Register of Legislation](https://www.legislation.gov.au/), [Federal Court of Australia](https://www.fedcourt.gov.au/digital-law-library/judgments/search), [High Court of Australia](https://eresources.hcourt.gov.au/), [NSW Caselaw](https://www.caselaw.nsw.gov.au/), [NSW Legislation](https://legislation.nsw.gov.au/), [Queensland Legislation](https://www.legislation.qld.gov.au/), [Western Australian Legislation](https://www.legislation.wa.gov.au/), [South Australian Legislation](https://www.legislation.sa.gov.au/) and [Tasmanian Legislation](https://www.legislation.tas.gov.au/) databases.

[`Inscriptis`](https://github.com/weblyzard/inscriptis) was used to extract the text of documents stored as HTML, [`pdfplumber`](https://github.com/jsvine/pdfplumber) for PDFs, [`striprtf`](https://github.com/joshy/striprtf) for RTFs and finally [`mammoth`](https://github.com/mwilliamson/python-mammoth) was used to convert DOCs and DOCXs to HTML before also extracting their text with `Inscriptis`.

The below table provides the date each source was last updated and the types of documents collected:
| Source | Date | Documents |
| --- | --- | --- |
| [Federal Register of Legislation](https://www.legislation.gov.au/) | 10 February 2024 | <ul><li>The most recent versions of all in force acts and the Constitution (primary legislation); and</li> <li>The most recent versions of all in force legislative instruments, notifiable instruments, administrative arrangements orders and prerogative instruments (secondary legislation).</li></ul> |
| [Federal Court of Australia](https://www.fedcourt.gov.au/digital-law-library/judgments/search) | 10 February 2024 | <ul><li>All decisions of the Federal Court of Australia, Industrial Relations Court of Australia, Australian Competition Tribunal, Copyright Tribunal, Defence Force Discipline Appeal Tribunal, Federal Police Disciplinary Tribunal, Trade Practices Tribunal and Supreme Court of Norfolk Island.</li></ul> |
| [High Court of Australia](https://eresources.hcourt.gov.au/) | 10 February 2024 | <ul><li>All decisions of the High Court of Australia.</li></ul> |
| [NSW Caselaw](https://www.caselaw.nsw.gov.au/) | 10 February 2024 | <ul><li>All decisions of the NSW Children's Court, Compensation Court, Court of Appeal, Court of Criminal Appeal, District Court, Drug Court, Industrial Relations Commission, Land and Environment Court, Local Court, Supreme Court, Administrative Decisions Tribunal, Civil and Administrative Tribunal, Dust Diseases Tribunal, Equal Opportunity Tribunal, Fair Trading Tribunal, Legal Services Tribunal, Medical Tribunal and Transport Appeals Boards.</li></ul> |
| [NSW Legislation](https://legislation.nsw.gov.au/) | 10 February 2024 | <ul><li>The most recent versions of all in force public and private acts (primary legislation); and</li> <li>The most recent versions of all in force statutory instruments and environmental planning instruments (secondary legislation).</li></ul> |
| [Queensland Legislation](https://www.legislation.qld.gov.au/) | 10 February 2024 | <ul><li>The most recent versions of all in force acts (primary legislation);</li> <li>The most recent versions of all in force statutory instruments (secondary legislation); and</li> <li>The as introduced versions of all bills.</li></ul> |
| [Western Australian Legislation](https://www.legislation.wa.gov.au/) | 10 February 2024 | <ul><li>The most recent versions of all in force acts (primary legislation); and</li> <li>The most recent versions of all in force subsidiary legislation (secondary legislation).</li></ul> |
| [South Australian Legislation](https://www.legislation.sa.gov.au/) | 10 February 2024 | <ul><li>The most recent versions of all in force acts (primary legislation); and</li> <li>The most recent versions of all in force proclamations, policies and regulations (secondary legislation).</li></ul> |
| [Tasmanian Legislation](https://www.legislation.tas.gov.au/) | 10 February 2024 | <ul><li>The most recent versions of all in force acts (primary legislation); and</li> <li>The most recent versions of all in force statutory rules (secondary legislation).</li></ul> |

The code used to create and update the Corpus can be found [here](https://github.com/umarbutler/open-australian-legal-corpus-creator).

Those interested in learning more about how the Corpus was built are encouraged to read Umar Butler's accompanying article, [*How I built the largest open database of Australian law*](https://umarbutler.com/how-i-built-the-largest-open-database-of-australian-law/).

## Changelog 🔄
All notable changes to the Corpus are documented in its [Changelog 🔄](https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus/blob/main/CHANGELOG.md).

This project adheres to [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) and [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## Licence 📜
As a work constituting a collection of documents that have been cleaned, structured, annotated and otherwise processed, the Corpus itself is licensed under the [Creative Commons Attribution 4.0 International Licence](https://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, on the condition that you give appropriate credit to the original author and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

Documents contained within the Corpus are distributed under relatively equally permissive licences that, in most cases, allow for both non-commercial and commercial usage and are available in the complete version of this licence [here](https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus/blob/main/LICENCE.md).

## Citation 🔖
If you've relied on the Corpus for your work, please cite:
```bibtex
@misc{butler-2024-open-australian-legal-corpus,
    author = {Butler, Umar},
    year = {2024},
    title = {Open Australian Legal Corpus},
    publisher = {Hugging Face},
    version = {4.2.1},
    doi = {10.57967/hf/1771},
    url = {https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus}
}
```

## Acknowledgements 🙏
In the spirit of reconciliation, the author acknowledges the Traditional Custodians of Country throughout Australia and their connections to land, sea and community. He pays his respect to their Elders past and present and extends that respect to all Aboriginal and Torres Strait Islander peoples today.

The author thanks the [Federal Register of Legislation](https://www.legislation.gov.au/), [Federal Court of Australia](https://www.fedcourt.gov.au/digital-law-library/judgments/search), [High Court of Australia](https://eresources.hcourt.gov.au/), [NSW Caselaw](https://www.caselaw.nsw.gov.au/), [NSW Legislation](https://legislation.nsw.gov.au/), [Queensland Legislation](https://www.legislation.qld.gov.au/), [Western Australian Legislation](https://www.legislation.wa.gov.au/), [South Australian Legislation](https://www.legislation.sa.gov.au/) and [Tasmanian Legislation](https://www.legislation.tas.gov.au/) for all granting him permission to scrape their data.

The author also acknowledges the creators of the many Python libraries relied upon in the creation of the Corpus, as well as the makers of the [Pile of Law](https://huggingface.co/datasets/pile-of-law/pile-of-law), which served as a great source of inspiration for this project.

Finally, the author is eternally grateful for the endless support of his wife and her willingness to put up with many a late night spent writing code and quashing bugs.
task_categories:text-generation task_categories:fill-mask task_categories:text-retrieval task_ids:language-modeling task_ids:masked-language-modeling task_ids:document-retrieval annotations_creators:no-annotation language_creators:found size_categories:100K<n<1M source_datasets:Federal Register of Legislation source_datasets:Federal Court of Australia source_datasets:High Court of Australia source_datasets:NSW Caselaw source_datasets:NSW Legislation source_datasets:Queensland Legislation source_datasets:Western Australian Legislation source_datasets:South Australian Legislation source_datasets:Tasmanian Legislation language:en license:other law legal australia doi:10.57967/hf/1771 region:us
{"annotations_creators": ["no-annotation"], "language_creators": ["found"], "language": ["en"], "license": "other", "size_categories": ["100K<n<1M"], "source_datasets": ["Federal Register of Legislation", "Federal Court of Australia", "High Court of Australia", "NSW Caselaw", "NSW Legislation", "Queensland Legislation", "Western Australian Legislation", "South Australian Legislation", "Tasmanian Legislation"], "task_categories": ["text-generation", "fill-mask", "text-retrieval"], "task_ids": ["language-modeling", "masked-language-modeling", "document-retrieval"], "pretty_name": "Open Australian Legal Corpus", "license_name": "open-australian-legal-corpus", "license_link": "https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus/blob/main/LICENCE.md", "tags": ["law", "legal", "australia"], "language_details": "en-AU, en-GB", "viewer": true, "dataset_info": {"config_name": "train", "features": [{"name": "version_id", "dtype": "string"}, {"name": "type", "dtype": "string"}, {"name": "jurisdiction", "dtype": "string"}, {"name": "source", "dtype": "string"}, {"name": "citation", "dtype": "string"}, {"name": "url", "dtype": "string"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 18296307089, "num_examples": 202260}], "download_size": 18365697826, "dataset_size": 18296307089}}
TAGS
#task_categories-text-generation #task_categories-fill-mask #task_categories-text-retrieval #task_ids-language-modeling #task_ids-masked-language-modeling #task_ids-document-retrieval #annotations_creators-no-annotation #language_creators-found #size_categories-100K<n<1M #source_datasets-Federal Register of Legislation #source_datasets-Federal Court of Australia #source_datasets-High Court of Australia #source_datasets-NSW Caselaw #source_datasets-NSW Legislation #source_datasets-Queensland Legislation #source_datasets-Western Australian Legislation #source_datasets-South Australian Legislation #source_datasets-Tasmanian Legislation #language-English #license-other #law #legal #australia #doi-10.57967/hf/1771 #region-us 

Open Australian Legal Corpus ‍️
===============================


<a href="URL alt="Release"><img src="URL


The Open Australian Legal Corpus is the first and only multijurisdictional open corpus of Australian legislative and judicial documents.


Comprised of 202,260 texts totalling over 50 million lines and 1.2 billion tokens, the Corpus includes every in force statute and regulation in the Commonwealth, New South Wales, Queensland, Western Australia, South Australia, Tasmania and Norfolk Island, in addition to thousands of bills and hundreds of thousands of court and tribunal decisions.


As the largest free and open database of its kind to date, the Corpus is intended to progress the burgeoning field of legal AI research in Australia by allowing researchers to pretrain and finetune machine learning models for downstream natural language processing tasks applied to the Australian legal domain such as document classification, summarisation, information retrieval and question answering.


To ensure its accessibility to as wide an audience as possible, the Corpus and all its documents are distributed under open source licences that, in most cases, allow for both non-commercial and commercial usage (see the Licence ).


Those interested in learning more about the Corpus are encouraged to read Umar Butler's accompanying article, *How I built the largest open database of Australian law*.


Usage ‍
-------


The below code snippet illustrates how the Corpus may be streamed or loaded with the Hugging Face Datasets Python library:


To speed up the loading of the Corpus, you may wish to install 'orjson'.


Statistics
----------


The Corpus is comprised of 202,260 documents, totalling 52,613,824 lines and 1,290,869,784 tokens.


A breakdown of the number of documents by type and source is provided below:



Structure ️
-----------


The Corpus is stored in URL, a json lines file where each line represents a document consisting of six keys:



Collection
----------


Documents were sourced from the Federal Register of Legislation, Federal Court of Australia, High Court of Australia, NSW Caselaw, NSW Legislation, Queensland Legislation, Western Australian Legislation, South Australian Legislation and Tasmanian Legislation databases.


'Inscriptis' was used to extract the text of documents stored as HTML, 'pdfplumber' for PDFs, 'striprtf' for RTFs and finally 'mammoth' was used to convert DOCs and DOCXs to HTML before also extracting their text with 'Inscriptis'.


The below table provides the date each source was last updated and the types of documents collected:


Source: Federal Register of Legislation, Date: 10 February 2024, Documents: * The most recent versions of all in force acts and the Constitution (primary legislation); and
* The most recent versions of all in force legislative instruments, notifiable instruments, administrative arrangements orders and prerogative instruments (secondary legislation).


Source: Federal Court of Australia, Date: 10 February 2024, Documents: * All decisions of the Federal Court of Australia, Industrial Relations Court of Australia, Australian Competition Tribunal, Copyright Tribunal, Defence Force Discipline Appeal Tribunal, Federal Police Disciplinary Tribunal, Trade Practices Tribunal and Supreme Court of Norfolk Island.


Source: High Court of Australia, Date: 10 February 2024, Documents: * All decisions of the High Court of Australia.


Source: NSW Caselaw, Date: 10 February 2024, Documents: * All decisions of the NSW Children's Court, Compensation Court, Court of Appeal, Court of Criminal Appeal, District Court, Drug Court, Industrial Relations Commission, Land and Environment Court, Local Court, Supreme Court, Administrative Decisions Tribunal, Civil and Administrative Tribunal, Dust Diseases Tribunal, Equal Opportunity Tribunal, Fair Trading Tribunal, Legal Services Tribunal, Medical Tribunal and Transport Appeals Boards.


Source: NSW Legislation, Date: 10 February 2024, Documents: * The most recent versions of all in force public and private acts (primary legislation); and
* The most recent versions of all in force statutory instruments and environmental planning instruments (secondary legislation).


Source: Queensland Legislation, Date: 10 February 2024, Documents: * The most recent versions of all in force acts (primary legislation);
* The most recent versions of all in force statutory instruments (secondary legislation); and
* The as introduced versions of all bills.


Source: Western Australian Legislation, Date: 10 February 2024, Documents: * The most recent versions of all in force acts (primary legislation); and
* The most recent versions of all in force subsidiary legislation (secondary legislation).


Source: South Australian Legislation, Date: 10 February 2024, Documents: * The most recent versions of all in force acts (primary legislation); and
* The most recent versions of all in force proclamations, policies and regulations (secondary legislation).


Source: Tasmanian Legislation, Date: 10 February 2024, Documents: * The most recent versions of all in force acts (primary legislation); and
* The most recent versions of all in force statutory rules (secondary legislation).



The code used to create and update the Corpus can be found here.


Those interested in learning more about how the Corpus was built are encouraged to read Umar Butler's accompanying article, *How I built the largest open database of Australian law*.


Changelog
---------


All notable changes to the Corpus are documented in its Changelog .


This project adheres to Keep a Changelog and Semantic Versioning.


Licence
-------


As a work constituting a collection of documents that have been cleaned, structured, annotated and otherwise processed, the Corpus itself is licensed under the Creative Commons Attribution 4.0 International Licence, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, on the condition that you give appropriate credit to the original author and the source, provide a link to the Creative Commons licence, and indicate if changes were made.


Documents contained within the Corpus are distributed under relatively equally permissive licences that, in most cases, allow for both non-commercial and commercial usage and are available in the complete version of this licence here.


If you've relied on the Corpus for your work, please cite:


Acknowledgements
----------------


In the spirit of reconciliation, the author acknowledges the Traditional Custodians of Country throughout Australia and their connections to land, sea and community. He pays his respect to their Elders past and present and extends that respect to all Aboriginal and Torres Strait Islander peoples today.


The author thanks the Federal Register of Legislation, Federal Court of Australia, High Court of Australia, NSW Caselaw, NSW Legislation, Queensland Legislation, Western Australian Legislation, South Australian Legislation and Tasmanian Legislation for all granting him permission to scrape their data.


The author also acknowledges the creators of the many Python libraries relied upon in the creation of the Corpus, as well as the makers of the Pile of Law, which served as a great source of inspiration for this project.


Finally, the author is eternally grateful for the endless support of his wife and her willingness to put up with many a late night spent writing code and quashing bugs.



TAGS
#task_categories-text-generation #task_categories-fill-mask #task_categories-text-retrieval #task_ids-language-modeling #task_ids-masked-language-modeling #task_ids-document-retrieval #annotations_creators-no-annotation #language_creators-found #size_categories-100K<n<1M #source_datasets-Federal Register of Legislation #source_datasets-Federal Court of Australia #source_datasets-High Court of Australia #source_datasets-NSW Caselaw #source_datasets-NSW Legislation #source_datasets-Queensland Legislation #source_datasets-Western Australian Legislation #source_datasets-South Australian Legislation #source_datasets-Tasmanian Legislation #language-English #license-other #law #legal #australia #doi-10.57967/hf/1771 #region-us 
