# Dataset Card for NbAiLab/NCC


## Table of Contents
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Data Fields](#data-fiels)
- [Dataset Creation](#dataset-creation)
- [Statistics](#statistics)
- [Document Types](#document-types)
- [Languages](#languages)
- [Publish Periode](#publish-periode)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Social Impact of Dataset](#social-impact-of-dataset)
- [Discussion of Biases](#discussion-of-biases)
- [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
- [Dataset Curators](#dataset-curators)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)

## Dataset Description
- **Homepage:** https://github.com/NbAiLab/notram
- **Repository:** https://github.com/NbAiLab/notram
- **Paper:** https://arxiv.org/abs/2104.09617
- **Point of Contact:** [Freddy Wetjen](mailto:freddy.wetjen@nb.no)

The Norwegian Colossal Corpus is a collection of multiple smaller Norwegian corpuses suitable for training large language models. We have done extensive cleaning on the datasets, and have made them available in a common format. The total size of the NCC is currently 45GB. 

## How to Use
```python
from datasets import load_dataset
data = load_dataset("NbAiLab/NCC", streaming=True)
```
## Download Data
If you do not want to use the HuggingFace Dataset-library for training, or if you want to do additional pre-processing, it is also possible to download the files locally.
```bash
# Clone the training set
git clone https://huggingface.co/datasets/NbAiLab/NCC

# Create one large training file of all shards without unpacking
cat NCC/data/train*.gz > onefile.json.gz
```

<details>
<summary>List of all the files.</summary>


* [train-shard-0001-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0001-of-0046.json.gz)
* [train-shard-0002-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0002-of-0046.json.gz)
* [train-shard-0003-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0003-of-0046.json.gz)
* [train-shard-0004-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0004-of-0046.json.gz)
* [train-shard-0005-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0005-of-0046.json.gz)
* [train-shard-0006-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0006-of-0046.json.gz)
* [train-shard-0007-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0007-of-0046.json.gz)
* [train-shard-0008-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0008-of-0046.json.gz)
* [train-shard-0009-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0009-of-0046.json.gz)
* [train-shard-0010-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0010-of-0046.json.gz)
* [train-shard-0011-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0011-of-0046.json.gz)
* [train-shard-0012-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0012-of-0046.json.gz)
* [train-shard-0013-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0013-of-0046.json.gz)
* [train-shard-0014-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0014-of-0046.json.gz)
* [train-shard-0015-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0015-of-0046.json.gz)
* [train-shard-0016-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0016-of-0046.json.gz)
* [train-shard-0017-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0017-of-0046.json.gz)
* [train-shard-0018-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0018-of-0046.json.gz)
* [train-shard-0019-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0019-of-0046.json.gz)
* [train-shard-0020-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0020-of-0046.json.gz)
* [train-shard-0021-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0021-of-0046.json.gz)
* [train-shard-0022-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0022-of-0046.json.gz)
* [train-shard-0023-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0023-of-0046.json.gz)
* [train-shard-0024-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0024-of-0046.json.gz)
* [train-shard-0025-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0025-of-0046.json.gz)
* [train-shard-0026-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0026-of-0046.json.gz)
* [train-shard-0027-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0027-of-0046.json.gz)
* [train-shard-0028-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0028-of-0046.json.gz)
* [train-shard-0029-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0029-of-0046.json.gz)
* [train-shard-0030-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0030-of-0046.json.gz)
* [train-shard-0031-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0031-of-0046.json.gz)
* [train-shard-0032-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0032-of-0046.json.gz)
* [train-shard-0033-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0033-of-0046.json.gz)
* [train-shard-0034-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0034-of-0046.json.gz)
* [train-shard-0035-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0035-of-0046.json.gz)
* [train-shard-0036-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0036-of-0046.json.gz)
* [train-shard-0037-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0037-of-0046.json.gz)
* [train-shard-0038-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0038-of-0046.json.gz)
* [train-shard-0039-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0039-of-0046.json.gz)
* [train-shard-0040-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0040-of-0046.json.gz)
* [train-shard-0041-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0041-of-0046.json.gz)
* [train-shard-0042-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0042-of-0046.json.gz)
* [train-shard-0043-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0043-of-0046.json.gz)
* [train-shard-0044-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0044-of-0046.json.gz)
* [train-shard-0045-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0045-of-0046.json.gz)
* [train-shard-0046-of-0046](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/train-shard-0046-of-0046.json.gz)
* [validation-shard-0001-of-0001](https://huggingface.co/datasets/NbAiLab/NCC/resolve/main/data/validation-shard-0001-of-0001.json.gz)


</details>

### Dataset Summary
The NCC dataset contains json lines with language training data. Here is an example json line:
```json
{
"id": "1006205",
"doc_type": "cc100",
"publish_year": 2021,
"lang_fasttext": "nn",
"lang_fasttext_conf": "0.641",
"text": "Eg har ein PLAN! KOS deg og ha ei fin helg"
}
```
## Data Fields
|**id:** | String with id to source of line and a unique identifier|
|:-----------|:------------|
|**doc_type** | String describing type of media text extracted from (I.e. book,newspaper etc)|
|**publish_year** | Integer. The year text published. When year is undetermined it is set to 2021.|
|**lang_fasttext** | String. Language of text identified by FastText|
|**lang_fasttext_conf** | String. Confidence calculated by FastText|
|**text** | String. The complete utf-8 document. If longer than 1M characters it is split.|

### Dataset Creation
We are providing a **train** and a **validation** split. The standard size of the validation is a single 1GB file, while train is sharded in 1GB chunks.
All files are gzipped.

Build date: 21012022

#### Initial Data Collection and Curation
The procedure for the dataset creation is described in detail in our paper.

### Summary
| Words         | Documents   |   Words/Document |
|--------------:|------------:|-----------------:|
| 6,905,570,165 | 20,830,348  |              331 |

### Document Types
| Source                                | Words         | Documents   | Words/Document   |
|--------------------------------------:|--------------:|------------:|-----------------:|
| newspaper_ocr                         | 1,974,452,883 | 9,872,470   | 199              |
| parliament                            | 1,273,353,169 | 9,321       | 136,611          |
| books                                 | 842,936,050   | 23,708      | 35,554           |
| newspapers_online_nb                  | 487,189,627   | 3,446,348   | 141              |
| maalfrid_regjeringen                  | 360,349,242   | 919,902     | 391              |
| maalfrid_ssb                          | 279,732,847   | 851,982     | 328              |
| maalfrid_uio                          | 181,916,296   | 771,480     | 235              |
| government_nb                         | 134,127,104   | 3,476       | 38,586           |
| wikipedia_download_nbo                | 110,845,615   | 523,593     | 211              |
| maalfrid_fylkesmannen                 | 102,849,898   | 463,021     | 222              |
| publicreports                         | 78,347,879    | 3,298       | 23,756           |
| maalfrid_nve                          | 66,656,315    | 301,966     | 220              |
| maalfrid_patentstyret                 | 64,985,154    | 213,991     | 303              |
| maalfrid_ntnu                         | 57,803,460    | 199,307     | 290              |
| newspapers_online_nn                  | 42,205,558    | 167,347     | 252              |
| lovdata_cd_odelsting_2005             | 36,370,948    | 1,933       | 18,815           |
| maalfrid_vegvesen                     | 33,431,887    | 166,203     | 201              |
| maalfrid_fhi                          | 32,784,098    | 144,363     | 227              |
| maalfrid_norad                        | 32,720,034    | 93,097      | 351              |
| maalfrid_skatteetaten                 | 32,567,691    | 82,589      | 394              |
| maalfrid_uib                          | 28,425,322    | 115,729     | 245              |
| wikipedia_download_nno                | 27,061,858    | 143,265     | 188              |
| maalfrid_forskningsradet              | 24,076,984    | 73,368      | 328              |
| maalfrid_nasjonalparkstyre            | 21,309,995    | 93,871      | 227              |
| government_nn                         | 18,316,345    | 1,063       | 17,230           |
| maalfrid_nmbu                         | 18,082,476    | 69,719      | 259              |
| maalfrid_oslomet                      | 17,710,771    | 47,022      | 376              |
| maalfrid_domstol                      | 16,678,270    | 51,038      | 326              |
| maalfrid_banenor                      | 16,445,420    | 70,360      | 233              |
| maalfrid_nav                          | 16,272,635    | 74,101      | 219              |
| maalfrid_landbruksdirektoratet        | 13,119,567    | 47,983      | 273              |
| maalfrid_helsedirektoratet            | 13,008,787    | 49,344      | 263              |
| maalfrid_nokut                        | 10,101,424    | 38,552      | 262              |
| maalfrid_hi                           | 10,046,751    | 39,065      | 257              |
| maalfrid_norges-bank                  | 9,924,489     | 37,171      | 266              |
| maalfrid_udir                         | 9,868,345     | 38,736      | 254              |
| maalfrid_vkm                          | 9,824,529     | 32,230      | 304              |
| maalfrid_nbim                         | 9,629,725     | 18,131      | 531              |
| maalfrid_miljodirektoratet            | 9,496,631     | 34,711      | 273              |
| maalfrid_distriktssenteret            | 9,375,506     | 38,525      | 243              |
| maalfrid_ngu                          | 9,231,905     | 34,619      | 266              |
| maalfrid_ptil                         | 9,214,434     | 34,250      | 269              |
| maalfrid_nord                         | 8,992,352     | 44,800      | 200              |
| maalfrid_fiskeridir                   | 8,297,897     | 33,446      | 248              |
| maalfrid_hivolda                      | 7,820,709     | 26,473      | 295              |
| maalfrid_difi                         | 7,789,290     | 35,733      | 217              |
| maalfrid_mattilsynet                  | 7,492,831     | 27,002      | 277              |
| maalfrid_havarikommisjonen            | 7,440,410     | 24,989      | 297              |
| maalfrid_kulturradet                  | 7,196,423     | 22,437      | 320              |
| maalfrid_ks                           | 6,915,503     | 27,439      | 252              |
| maalfrid_kystverket                   | 6,713,070     | 30,975      | 216              |
| maalfrid_udi                          | 6,433,540     | 19,134      | 336              |
| maalfrid_uia                          | 5,964,644     | 23,861      | 249              |
| maalfrid_hjelpemiddeldatabasen        | 5,892,662     | 34,192      | 172              |
| maalfrid_khrono                       | 5,859,186     | 19,970      | 293              |
| maalfrid_helsetilsynet                | 5,803,000     | 18,365      | 315              |
| maalfrid_moreforsk                    | 5,622,025     | 21,579      | 260              |
| maalfrid_jernbanedirektoratet         | 5,461,268     | 21,666      | 252              |
| maalfrid_veiviseren                   | 5,316,521     | 18,026      | 294              |
| lovdata_cd_somb_rundskriv_2005        | 5,264,746     | 3,215       | 1,637            |
| maalfrid_dsb                          | 5,199,259     | 17,814      | 291              |
| lovdata_cd_sentrale_forskrifter_2005  | 5,037,694     | 11,467      | 439              |
| maalfrid_husbanken                    | 4,711,069     | 15,053      | 312              |
| maalfrid_legemiddelverket             | 4,689,988     | 20,192      | 232              |
| maalfrid_vetinst                      | 4,674,951     | 14,492      | 322              |
| maalfrid_imdi                         | 4,636,355     | 15,290      | 303              |
| maalfrid_forsvarsbygg                 | 4,567,318     | 18,886      | 241              |
| maalfrid_sdir                         | 4,540,110     | 15,202      | 298              |
| maalfrid_konkurransetilsynet          | 4,512,807     | 12,617      | 357              |
| maalfrid_dsa                          | 4,498,837     | 15,898      | 282              |
| maalfrid_arkivverket                  | 4,493,280     | 16,515      | 272              |
| maalfrid_hiof                         | 4,473,731     | 23,119      | 193              |
| maalfrid_ehelse                       | 4,379,984     | 22,553      | 194              |
| maalfrid_inn                          | 4,326,704     | 26,277      | 164              |
| maalfrid_klagenemndssekretariatet     | 4,181,685     | 11,916      | 350              |
| maalfrid_sprakradet                   | 4,097,815     | 15,187      | 269              |
| maalfrid_dibk                         | 3,967,428     | 15,509      | 255              |
| maalfrid_nhh                          | 3,962,033     | 15,678      | 252              |
| maalfrid_kartverket                   | 3,732,184     | 18,710      | 199              |
| maalfrid_riksrevisjonen               | 3,680,555     | 10,922      | 336              |
| maalfrid_toll                         | 3,510,061     | 13,777      | 254              |
| maalfrid_nibio                        | 3,456,026     | 17,104      | 202              |
| maalfrid_met                          | 3,446,762     | 18,282      | 188              |
| maalfrid_bufdir                       | 3,354,740     | 11,470      | 292              |
| maalfrid_artsdatabanken               | 3,193,511     | 9,009       | 354              |
| maalfrid_politiet                     | 3,167,395     | 10,501      | 301              |
| maalfrid_nkom                         | 3,127,687     | 10,002      | 312              |
| maalfrid_vestlandfylke                | 3,060,166     | 12,075      | 253              |
| maalfrid_uis                          | 2,924,821     | 9,838       | 297              |
| maalfrid_sykkelbynettverket           | 2,820,702     | 11,818      | 238              |
| maalfrid_nlr                          | 2,646,014     | 15,851      | 166              |
| maalfrid_seniorporten                 | 2,616,054     | 8,111       | 322              |
| maalfrid_npd                          | 2,597,831     | 10,742      | 241              |
| maalfrid_aldringoghelse               | 2,430,767     | 6,788       | 358              |
| maalfrid_custompublish                | 2,430,747     | 9,184       | 264              |
| maalfrid_bioteknologiradet            | 2,393,891     | 5,996       | 399              |
| maalfrid_arbeidstilsynet              | 2,379,597     | 6,882       | 345              |
| maalfrid_nyemetoder                   | 2,376,468     | 10,771      | 220              |
| maalfrid_riksantikvaren               | 2,257,491     | 8,756       | 257              |
| maalfrid_sjt                          | 2,238,168     | 11,189      | 200              |
| lovdata_cd_lokaleforskrifter_2005     | 2,176,221     | 22,274      | 97               |
| maalfrid_hvl                          | 2,149,292     | 9,395       | 228              |
| maalfrid_luftfartstilsynet            | 2,101,272     | 9,866       | 212              |
| maalfrid_dfo                          | 2,073,203     | 9,165       | 226              |
| maalfrid_ldo                          | 2,047,969     | 7,299       | 280              |
| maalfrid_kompetansenorge              | 1,952,035     | 10,245      | 190              |
| maalfrid_forbrukerradet               | 1,945,089     | 7,330       | 265              |
| maalfrid_himolde                      | 1,913,699     | 9,975       | 191              |
| maalfrid_usn                          | 1,793,297     | 7,403       | 242              |
| lovdata_cd_norgeslover_2005           | 1,760,884     | 1,386       | 1,270            |
| maalfrid_naku                         | 1,754,510     | 5,239       | 334              |
| maalfrid_medietilsynet                | 1,608,424     | 6,611       | 243              |
| maalfrid_matematikksenteret           | 1,567,505     | 7,298       | 214              |
| maalfrid_forskningsetikk              | 1,545,336     | 5,545       | 278              |
| maalfrid_diku                         | 1,542,929     | 6,241       | 247              |
| maalfrid_godeidrettsanlegg            | 1,506,577     | 6,115       | 246              |
| maalfrid_dirmin                       | 1,467,255     | 5,303       | 276              |
| maalfrid_diskrimineringsnemnda        | 1,463,291     | 4,168       | 351              |
| maalfrid_naturfag                     | 1,450,662     | 5,976       | 242              |
| maalfrid_arbeidsretten                | 1,440,074     | 4,754       | 302              |
| lovdata_cd_rtv_rundskriv_2005         | 1,366,872     | 9,596       | 142              |
| maalfrid_fellesstudentsystem          | 1,359,292     | 10,321      | 131              |
| maalfrid_nupi                         | 1,286,395     | 5,491       | 234              |
| maalfrid_kriminalitetsforebygging     | 1,201,477     | 4,667       | 257              |
| maalfrid_anskaffelser                 | 1,187,544     | 5,479       | 216              |
| maalfrid_folketrygdfondet             | 1,183,502     | 4,253       | 278              |
| maalfrid_miljopakken                  | 1,170,252     | 5,513       | 212              |
| maalfrid_nih                          | 1,116,791     | 5,271       | 211              |
| maalfrid_statsbygg                    | 1,103,635     | 4,439       | 248              |
| lovdata_cd_skatt_rundskriv_2005       | 1,102,142     | 398         | 2,769            |
| maalfrid_nb                           | 1,055,200     | 4,135       | 255              |
| maalfrid_npolar                       | 1,051,181     | 2,653       | 396              |
| maalfrid_unit                         | 1,049,621     | 6,329       | 165              |
| maalfrid_valgdirektoratet             | 1,009,941     | 9,131       | 110              |
| maalfrid_barneombudet                 | 980,751       | 2,807       | 349              |
| maalfrid_datatilsynet                 | 974,679       | 2,965       | 328              |
| maalfrid_lottstift                    | 959,590       | 3,578       | 268              |
| maalfrid_aho                          | 953,568       | 4,528       | 210              |
| maalfrid_sykehuspartner               | 939,625       | 4,579       | 205              |
| maalfrid_naturfagsenteret             | 897,049       | 3,859       | 232              |
| maalfrid_khio                         | 849,973       | 3,377       | 251              |
| maalfrid_spesialenheten               | 824,209       | 2,127       | 387              |
| maalfrid_xn--miljlftet-o8ab           | 803,011       | 3,384       | 237              |
| maalfrid_samordnaopptak               | 792,595       | 2,368       | 334              |
| maalfrid_helsenorge                   | 780,465       | 3,034       | 257              |
| maalfrid_skrivesenteret               | 777,204       | 4,161       | 186              |
| maalfrid_mareano                      | 760,645       | 3,724       | 204              |
| maalfrid_fiskeridirektoratet          | 755,997       | 2,444       | 309              |
| maalfrid_sykehusinnkjop               | 738,720       | 4,340       | 170              |
| maalfrid_matportalen                  | 630,990       | 2,370       | 266              |
| maalfrid_spk                          | 613,180       | 2,152       | 284              |
| maalfrid_justervesenet                | 595,014       | 1,904       | 312              |
| maalfrid_pasientsikkerhetsprogrammet  | 594,399       | 4,684       | 126              |
| maalfrid_nhn                          | 579,713       | 3,581       | 161              |
| maalfrid_sshf                         | 572,570       | 1,897       | 301              |
| maalfrid_bibliotekutvikling           | 560,126       | 3,216       | 174              |
| maalfrid_nysgjerrigper                | 559,207       | 3,019       | 185              |
| maalfrid_nodnett                      | 538,021       | 2,689       | 200              |
| maalfrid_une                          | 513,586       | 1,255       | 409              |
| maalfrid_giek                         | 512,569       | 1,796       | 285              |
| maalfrid_samas                        | 501,177       | 2,548       | 196              |
| maalfrid_kriminalomsorgen             | 496,062       | 1,951       | 254              |
| maalfrid_kjonnsforskning              | 483,376       | 1,426       | 338              |
| maalfrid_kunstkultursenteret          | 470,009       | 1,435       | 327              |
| lovdata_cd_rundskriv_lovavdeling_2005 | 469,295       | 405         | 1,158            |
| maalfrid_nynorsksenteret              | 460,165       | 2,085       | 220              |
| maalfrid_ceres                        | 448,920       | 1,950       | 230              |
| maalfrid_stami                        | 445,031       | 1,160       | 383              |
| maalfrid_nsm                          | 442,110       | 1,536       | 287              |
| maalfrid_gjenopptakelse               | 420,205       | 1,467       | 286              |
| maalfrid_nfi                          | 420,128       | 1,523       | 275              |
| maalfrid_nidsenter                    | 410,785       | 1,631       | 251              |
| maalfrid_nasjonalmuseet               | 390,036       | 1,087       | 358              |
| maalfrid_forbrukertilsynet            | 387,579       | 1,227       | 315              |
| maalfrid_natursekken                  | 378,442       | 3,563       | 106              |
| maalfrid_fordelingsutvalget           | 355,121       | 1,385       | 256              |
| maalfrid_digdir                       | 349,548       | 2,105       | 166              |
| maalfrid_forsvaret                    | 331,183       | 1,215       | 272              |
| maalfrid_beccle                       | 329,568       | 1,517       | 217              |
| maalfrid_romsenter                    | 329,304       | 1,133       | 290              |
| maalfrid_geonorge                     | 301,869       | 1,622       | 186              |
| maalfrid_universell                   | 263,740       | 2,155       | 122              |
| maalfrid_ovf                          | 262,542       | 930         | 282              |
| maalfrid_forbrukereuropa              | 259,420       | 1,018       | 254              |
| maalfrid_politihogskolen              | 258,615       | 1,229       | 210              |
| maalfrid_vinmonopolet                 | 245,685       | 671         | 366              |
| maalfrid_energimerking                | 237,243       | 1,033       | 229              |
| maalfrid_ombudsmann                   | 225,947       | 418         | 540              |
| maalfrid_vea-fs                       | 224,712       | 1,261       | 178              |
| maalfrid_traumebevisst                | 224,297       | 2,417       | 92               |
| maalfrid_npe                          | 205,102       | 1,000       | 205              |
| maalfrid_pkh                          | 201,503       | 791         | 254              |
| maalfrid_helfo                        | 193,880       | 988         | 196              |
| maalfrid_opplaringslovutvalget        | 193,590       | 549         | 352              |
| maalfrid_regionaleforskningsfond      | 187,261       | 989         | 189              |
| maalfrid_nafkam                       | 177,295       | 571         | 310              |
| maalfrid_jernbanemagasinet            | 174,152       | 412         | 422              |
| maalfrid_polarhistorie                | 171,386       | 382         | 448              |
| maalfrid_aasentunet                   | 161,626       | 529         | 305              |
| maalfrid_riksteatret                  | 159,991       | 798         | 200              |
| maalfrid_realfagsloyper               | 157,166       | 748         | 210              |
| maalfrid_koro                         | 153,304       | 574         | 267              |
| maalfrid_squarespace                  | 146,931       | 504         | 291              |
| maalfrid_politietssikkerhetstjeneste  | 143,781       | 469         | 306              |
| maalfrid_unknown                      | 139,263       | 700         | 198              |
| maalfrid_whocc                        | 121,616       | 656         | 185              |
| maalfrid_konfliktraadet               | 120,258       | 372         | 323              |
| maalfrid_okokrim                      | 115,842       | 372         | 311              |
| maalfrid_brreg                        | 112,787       | 571         | 197              |
| maalfrid_riksmekleren                 | 110,737       | 558         | 198              |
| maalfrid_sismo                        | 110,700       | 309         | 358              |
| maalfrid_radetfordyreetikk            | 99,241        | 441         | 225              |
| maalfrid_akkreditert                  | 99,040        | 503         | 196              |
| maalfrid_sivilforsvaret               | 97,679        | 514         | 190              |
| maalfrid_lanekassen                   | 95,286        | 301         | 316              |
| maalfrid_digidel                      | 95,140        | 607         | 156              |
| maalfrid_generaladvokaten             | 91,385        | 294         | 310              |
| maalfrid_uit                          | 90,273        | 602         | 149              |
| maalfrid_nyinorge                     | 88,466        | 199         | 444              |
| maalfrid_lokforerskolen               | 87,224        | 468         | 186              |
| maalfrid_varsom                       | 85,382        | 563         | 151              |
| maalfrid_ffi                          | 80,137        | 220         | 364              |
| maalfrid_kulturminnefondet            | 79,767        | 411         | 194              |
| maalfrid_unesco                       | 76,951        | 382         | 201              |
| maalfrid_yrkesfisker                  | 74,807        | 501         | 149              |
| maalfrid_dekom                        | 72,148        | 1,307       | 55               |
| maalfrid_omsorgsforskning             | 71,675        | 321         | 223              |
| maalfrid_lektor2                      | 67,385        | 549         | 122              |
| maalfrid_openaccess                   | 63,554        | 192         | 331              |
| maalfrid_ssn                          | 63,036        | 302         | 208              |
| maalfrid_lokalhistorie                | 59,854        | 241         | 248              |
| maalfrid_nlb                          | 57,872        | 200         | 289              |
| maalfrid_riksadvokaten                | 57,563        | 155         | 371              |
| maalfrid_laudim                       | 57,500        | 393         | 146              |
| maalfrid_denkulturelleskolesekken     | 46,018        | 243         | 189              |
| maalfrid_sivilrett                    | 44,062        | 142         | 310              |
| maalfrid_htu                          | 43,330        | 169         | 256              |
| maalfrid_yr                           | 40,646        | 562         | 72               |
| maalfrid_informasjonskompetanse       | 40,351        | 330         | 122              |
| maalfrid_dep                          | 38,882        | 126         | 308              |
| maalfrid_finansportalen               | 38,506        | 180         | 213              |
| maalfrid_feide                        | 36,715        | 267         | 137              |
| maalfrid_kulturped                    | 36,013        | 96          | 375              |
| maalfrid_fug                          | 34,158        | 120         | 284              |
| maalfrid_kulturoghelse                | 33,424        | 184         | 181              |
| maalfrid_helseklage                   | 32,756        | 124         | 264              |
| maalfrid_nbsk                         | 30,674        | 211         | 145              |
| maalfrid_matogindustri                | 29,922        | 194         | 154              |
| maalfrid_sinn                         | 27,541        | 150         | 183              |
| maalfrid_transport21                  | 25,317        | 90          | 281              |
| maalfrid_konkursradet                 | 23,505        | 76          | 309              |
| maalfrid_vergemal                     | 23,271        | 77          | 302              |
| maalfrid_norec                        | 22,496        | 78          | 288              |
| maalfrid_pts                          | 20,459        | 78          | 262              |
| maalfrid_nasjonaleturistveger         | 19,922        | 110         | 181              |
| maalfrid_iearth                       | 19,281        | 146         | 132              |
| maalfrid_hjelpelinjen                 | 19,209        | 85          | 225              |
| maalfrid_russamtalen                  | 17,999        | 65          | 276              |
| maalfrid_xn--kvinneligomskjring-1ub   | 17,701        | 77          | 229              |
| maalfrid_nynorskbok                   | 17,600        | 96          | 183              |
| maalfrid_regjeringsadvokaten          | 17,416        | 55          | 316              |
| maalfrid_memu                         | 17,311        | 98          | 176              |
| maalfrid_xn--tilbakefring-2jb         | 15,814        | 49          | 322              |
| maalfrid_xn--forskerfr-t8a            | 15,724        | 172         | 91               |
| maalfrid_ringerikefengsel             | 15,669        | 28          | 559              |
| maalfrid_skeivtarkiv                  | 15,537        | 69          | 225              |
| maalfrid_samfunnskunnskap             | 15,110        | 60          | 251              |
| maalfrid_fordelingsutvalet            | 15,017        | 34          | 441              |
| maalfrid_skattefunn                   | 14,599        | 51          | 286              |
| maalfrid_shiprep                      | 14,165        | 142         | 99               |
| maalfrid_haldenfengsel                | 13,625        | 37          | 368              |
| maalfrid_sevuppt                      | 13,332        | 52          | 256              |
| maalfrid_forbrukerklageutvalget       | 12,698        | 49          | 259              |
| maalfrid_mhfa                         | 11,999        | 144         | 83               |
| maalfrid_ah                           | 11,787        | 36          | 327              |
| maalfrid_nettvett                     | 11,002        | 43          | 255              |
| maalfrid_uh-it                        | 10,828        | 273         | 39               |
| maalfrid_fishgen                      | 10,199        | 28          | 364              |
| maalfrid_designavgang                 | 10,164        | 75          | 135              |
| maalfrid_global                       | 9,051         | 41          | 220              |
| maalfrid_havmiljo                     | 8,607         | 68          | 126              |
| maalfrid_valg                         | 8,516         | 47          | 181              |
| maalfrid_miljoklagenemnda             | 7,797         | 35          | 222              |
| maalfrid_altinn                       | 7,695         | 49          | 157              |
| maalfrid_spinn-inn                    | 7,674         | 47          | 163              |
| maalfrid_kantinekurset                | 7,217         | 53          | 136              |
| maalfrid_bastoyfengsel                | 7,142         | 56          | 127              |
| maalfrid_norskpetroleum               | 6,083         | 119         | 51               |
| maalfrid_voldsoffererstatning         | 5,827         | 26          | 224              |
| maalfrid_musikkbasertmiljobehandling  | 5,186         | 39          | 132              |
| maalfrid_prosjektveiviseren           | 5,019         | 14          | 358              |
| maalfrid_aldersvennlig                | 4,919         | 32          | 153              |
| maalfrid_barentswatch                 | 4,829         | 32          | 150              |
| maalfrid_fmfiavo@fylkesmannen         | 4,702         | 68          | 69               |
| maalfrid_kk-utvalget                  | 4,697         | 19          | 247              |
| maalfrid_agropub                      | 4,434         | 17          | 260              |
| maalfrid_utdanningiverden             | 4,266         | 13          | 328              |
| maalfrid_overgangsbolig               | 3,769         | 35          | 107              |
| maalfrid_forsvaretsmuseer             | 3,706         | 34          | 109              |
| maalfrid_okopark                      | 3,282         | 12          | 273              |
| maalfrid_pst                          | 2,866         | 14          | 204              |
| maalfrid_sikkerhverdag                | 2,697         | 18          | 149              |
| maalfrid_arkitektur                   | 2,436         | 15          | 162              |
| maalfrid_velgekte                     | 2,287         | 10          | 228              |
| maalfrid_addlab                       | 2,109         | 12          | 175              |
| maalfrid_romerikefengsel              | 2,088         | 19          | 109              |
| maalfrid_utdanning                    | 2,009         | 12          | 167              |
| maalfrid_grunderskolen                | 1,994         | 7           | 284              |
| maalfrid_umb                          | 1,934         | 8           | 241              |
| maalfrid_oslofengsel                  | 1,756         | 8           | 219              |
| maalfrid_hjorteviltregisteret         | 1,600         | 5           | 320              |
| maalfrid_alleteller                   | 1,511         | 7           | 215              |
| maalfrid_webhuset                     | 1,409         | 5           | 281              |
| maalfrid_lykillinn                    | 1,349         | 4           | 337              |
| maalfrid_kulturfag                    | 1,215         | 6           | 202              |
| maalfrid_unimus                       | 940           | 4           | 235              |
| maalfrid_anleggsregisteret            | 928           | 5           | 185              |
| maalfrid_mangfoldsprisen              | 597           | 3           | 199              |
| maalfrid_algae2future                 | 456           | 8           | 57               |
| maalfrid_mammapresenterer             | 447           | 2           | 223              |
| maalfrid_karriereveiledning           | 391           | 27          | 14               |
| maalfrid_nodsms                       | 351           | 4           | 87               |
| maalfrid_kildekompasset               | 302           | 1           | 302              |
| maalfrid_praksisfou                   | 297           | 1           | 297              |
| maalfrid_retttilaalese                | 246           | 3           | 82               |
| maalfrid_indreostfoldfengsel          | 215           | 3           | 71               |
| maalfrid_xn--kroppsvingsforskning-gcc | 205           | 2           | 102              |
| maalfrid_pahoyden                     | 154           | 1           | 154              |
| maalfrid_norren                       | 42            | 1           | 42               |

### Languages
| Language   | Words         | Documents   | Words/Document   |
|-----------:|--------------:|------------:|-----------------:|
| no         | 5,050,752,505 | 17,177,223  | 294              |
| da         | 940,216,574   | 574,211     | 1,637            |
| en         | 474,855,361   | 1,526,795   | 311              |
| nn         | 299,753,996   | 987,701     | 303              |
| fr         | 49,409,701    | 108,071     | 457              |
| de         | 27,159,878    | 85,230      | 318              |
| sv         | 18,773,092    | 118,753     | 158              |
| es         | 10,057,791    | 42,177      | 238              |
| fi         | 8,104,322     | 46,710      | 173              |
| et         | 3,309,661     | 24,183      | 136              |
| cs         | 2,652,151     | 21,793      | 121              |
| pt         | 2,550,218     | 16,407      | 155              |
| oc         | 2,123,730     | 4,927       | 431              |
| nl         | 1,984,501     | 11,813      | 167              |
| zh         | 1,470,751     | 8,146       | 180              |
| uk         | 1,459,484     | 5,096       | 286              |
| ca         | 1,370,260     | 4,476       | 306              |
| it         | 1,293,230     | 8,479       | 152              |
| la         | 1,281,920     | 797         | 1,608            |
| ru         | 1,231,482     | 6,796       | 181              |
| pl         | 852,304       | 9,396       | 90               |
| eu         | 831,276       | 3,195       | 260              |
| hu         | 659,973       | 8,499       | 77               |
| fa         | 494,551       | 2,047       | 241              |
| ja         | 351,634       | 4,994       | 70               |
| is         | 309,422       | 1,207       | 256              |
| id         | 226,296       | 2,033       | 111              |
| ar         | 205,632       | 1,173       | 175              |
| sl         | 140,913       | 1,858       | 75               |
| vi         | 139,122       | 982         | 141              |
| so         | 128,303       | 592         | 216              |
| hr         | 124,033       | 1,081       | 114              |
| el         | 117,624       | 618         | 190              |
| lv         | 106,626       | 123         | 866              |
| tr         | 92,680        | 1,630       | 56               |
| ro         | 80,804        | 635         | 127              |
| sr         | 71,953        | 970         | 74               |
| lt         | 70,148        | 869         | 80               |
| gl         | 65,152        | 692         | 94               |
| war        | 56,369        | 274         | 205              |
| ko         | 56,057        | 1,006       | 55               |
| th         | 54,067        | 367         | 147              |
| am         | 44,818        | 317         | 141              |
| sk         | 39,416        | 1,000       | 39               |
| ml         | 35,575        | 156         | 228              |
| ceb        | 35,337        | 331         | 106              |
| sq         | 34,461        | 238         | 144              |
| tl         | 30,839        | 177         | 174              |
| kk         | 27,827        | 72          | 386              |
| eo         | 24,187        | 859         | 28               |
| mn         | 21,540        | 22          | 979              |
| sw         | 18,670        | 72          | 259              |
| pnb        | 18,403        | 80          | 230              |
| sh         | 17,807        | 213         | 83               |
| gu         | 16,973        | 13          | 1,305            |
| bg         | 16,495        | 100         | 164              |
| ur         | 15,650        | 169         | 92               |
| mk         | 13,305        | 65          | 204              |
| ckb        | 9,119         | 43          | 212              |
| ku         | 9,071         | 57          | 159              |
| ast        | 7,919         | 73          | 108              |
| az         | 7,907         | 59          | 134              |
| ms         | 7,051         | 483         | 14               |
| uz         | 6,924         | 56          | 123              |
| ta         | 4,180         | 60          | 69               |
| fy         | 3,841         | 68          | 56               |
| ga         | 3,761         | 174         | 21               |
| hy         | 3,456         | 43          | 80               |
| pa         | 3,299         | 17          | 194              |
| hi         | 2,783         | 39          | 71               |
| be         | 2,556         | 62          | 41               |
| bo         | 2,551         | 1           | 2,551            |
| ht         | 2,534         | 11          | 230              |
| jv         | 2,341         | 91          | 25               |
| min        | 2,206         | 18          | 122              |
| cy         | 2,052         | 52          | 39               |
| bs         | 2,047         | 66          | 31               |
| als        | 1,918         | 66          | 29               |
| su         | 1,888         | 29          | 65               |
| nds        | 1,869         | 162         | 11               |
| ps         | 1,832         | 15          | 122              |
| bn         | 1,797         | 22          | 81               |
| qu         | 1,498         | 14          | 107              |
| ilo        | 1,126         | 25          | 45               |
| mt         | 968           | 16          | 60               |
| si         | 942           | 29          | 32               |
| te         | 888           | 18          | 49               |
| my         | 784           | 15          | 52               |
| af         | 741           | 32          | 23               |
| io         | 715           | 15          | 47               |
| tt         | 684           | 22          | 31               |
| km         | 674           | 11          | 61               |
| br         | 645           | 40          | 16               |
| gn         | 638           | 11          | 58               |
| jbo        | 611           | 27          | 22               |
| as         | 584           | 2           | 292              |
| ug         | 581           | 6           | 96               |
| kv         | 562           | 3           | 187              |
| kn         | 544           | 22          | 24               |
| pam        | 480           | 2           | 240              |
| kw         | 475           | 19          | 25               |
| vep        | 419           | 34          | 12               |
| he         | 412           | 18          | 22               |
| ka         | 351           | 20          | 17               |
| yo         | 281           | 9           | 31               |
| wa         | 268           | 38          | 7                |
| ky         | 228           | 10          | 22               |
| azb        | 216           | 1           | 216              |
| ba         | 203           | 5           | 40               |
| gom        | 174           | 12          | 14               |
| ia         | 140           | 15          | 9                |
| mr         | 138           | 10          | 13               |
| lmo        | 134           | 27          | 4                |
| tg         | 129           | 3           | 43               |
| lb         | 115           | 26          | 4                |
| pms        | 115           | 16          | 7                |
| vec        | 67            | 3           | 22               |
| rue        | 67            | 2           | 33               |
| sco        | 61            | 6           | 10               |
| ie         | 59            | 11          | 5                |
| hsb        | 57            | 3           | 19               |
| ne         | 56            | 6           | 9                |
| bar        | 46            | 7           | 6                |
| cbk        | 46            | 2           | 23               |
| or         | 44            | 2           | 22               |
| mg         | 38            | 8           | 4                |
| os         | 36            | 3           | 12               |
| tk         | 36            | 4           | 9                |
| arz        | 31            | 1           | 31               |
| li         | 29            | 6           | 4                |
| gd         | 29            | 2           | 14               |
| eml        | 24            | 5           | 4                |
| diq        | 20            | 2           | 10               |
| lrc        | 20            | 1           | 20               |
| dsb        | 19            | 1           | 19               |
| yue        | 19            | 1           | 19               |
| nap        | 16            | 1           | 16               |
| nah        | 14            | 2           | 7                |
| wuu        | 14            | 1           | 14               |
| sd         | 14            | 1           | 14               |
| frr        | 13            | 3           | 4                |
| rm         | 12            | 2           | 6                |
| cv         | 12            | 1           | 12               |
| scn        | 9             | 2           | 4                |
| bh         | 8             | 1           | 8                |
| bcl        | 8             | 1           | 8                |
| co         | 7             | 1           | 7                |
| ce         | 4             | 1           | 4                |
| new        | 4             | 1           | 4                |
| vo         | 3             | 2           | 1                |
| mzn        | 3             | 1           | 3                |
| gv         | 3             | 1           | 3                |
| lo         | 2             | 1           | 2                |

### Publish Periode
|   Decade | Words         | Documents   | Words/Document   |
|---------:|--------------:|------------:|-----------------:|
|     2020 | 4,090,213,596 | 10,934,550  | 523              |
|     2010 | 355,391,417   | 2,415,563   | 1,511            |
|     2000 | 447,853,330   | 1,705,354   | 2,773            |
|     1990 | 767,392,364   | 2,513,364   | 3,051            |
|     1980 | 160,980,586   | 538,665     | 3,011            |
|     1970 | 186,113,674   | 829,646     | 2,222            |
|     1960 | 149,421,535   | 834,219     | 1,807            |
|     1950 | 97,863,608    | 478,628     | 2,041            |
|     1940 | 122,648,278   | 570,154     | 2,307            |
|     1930 | 35,635,053    | 697         | 508,420          |
|     1920 | 50,381,418    | 1,049       | 484,836          |
|     1910 | 62,599,984    | 1,221       | 504,678          |
|     1900 | 60,019,080    | 1,130       | 527,329          |
|     1890 | 86,781,861    | 1,777       | 485,878          |
|     1880 | 58,546,570    | 1,064       | 553,442          |
|     1870 | 26,492,662    | 632         | 407,191          |
|     1860 | 39,176,930    | 698         | 543,151          |
|     1850 | 53,801,490    | 846         | 634,038          |
|     1840 | 30,434,939    | 522         | 581,593          |
|     1830 | 18,189,838    | 368         | 481,719          |
|     1820 | 4,721,154     | 144         | 338,350          |
|     1810 | 910,798       | 57          | 124,880          |

## Considerations for Using the Data
This corpus contains data under copyright and is not allowed to be used outide the National Library of Norway. The dataset should not be distributed.

### Discussion of Biases
Please refer to our paper.

### Dataset Curators
[Freddy Wetjen](mailto:Freddy.wetjen@nb.no) and [Per Egil Kummervold](mailto:Per.Kummervold@nb.no)


## License
Various licences applies to different parts of the corpus. Every document in the corpus has a tag telling what **"doc_type"** it belongs to. If you are unable to accept any of the licenses, you should filter out the **"doc_type"** with a conflicting license. 

| Doc_type  | License  | 
| :-------- | :------------- |  
| government_nb, government_nn, parliament, publicreports, lovdata_cd_\*, maalfrid_\* | [NLOD 2.0](https://data.norge.no/nlod/en/2.0/)|
| newspapers_ocr, newspapers_pdf, books| [CC0 1.0](https://creativecommons.org/publicdomain/zero/1.0/)|
| newspapers_online_nb, newspapers_online_nn | [CC BY-NC 2.0](https://creativecommons.org/licenses/by-nc/2.0/)|
| opensubtitles, wikipedia | [CC BY-SA 3.0](https://creativecommons.org/licenses/by-sa/3.0/)
|

### Citation Information
We are preparing an article with detailed information about this corpus. Until it is published, please cite out paper discussing the first version of this corpus:
```
@inproceedings{kummervold-etal-2021-operationalizing,
title = {Operationalizing a National Digital Library: The Case for a {N}orwegian Transformer Model},
author = {Kummervold, Per E  and
De la Rosa, Javier  and
Wetjen, Freddy  and
Brygfjeld, Svein Arne",
booktitle = {Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa)},
year = "2021",
address = "Reykjavik, Iceland (Online)",
publisher = {Link{"o}ping University Electronic Press, Sweden},
url = "https://aclanthology.org/2021.nodalida-main.3",
pages = "20--29",
abstract = "In this work, we show the process of building a large-scale training set from digital and digitized collections at a national library.
The resulting Bidirectional Encoder Representations from Transformers (BERT)-based language model for Norwegian outperforms multilingual BERT (mBERT) models
in several token and sequence classification tasks for both Norwegian Bokm{aa}l and Norwegian Nynorsk. Our model also improves the mBERT performance for other
languages present in the corpus such as English, Swedish, and Danish. For languages not included in the corpus, the weights degrade moderately while keeping strong multilingual properties. Therefore,
we show that building high-quality models within a memory institution using somewhat noisy optical character recognition (OCR) content is feasible, and we hope to pave the way for other memory institutions to follow.",
}
```

task_categories:text-generation task_ids:language-modeling annotations_creators:no-annotation language_creators:found multilinguality:multilingual size_categories:2G<n<1B source_datasets:original language:en language:nb language:no language:nn language:sv language:da language:is language:fo license:other arxiv:2104.09617 region:us
{"annotations_creators": ["no-annotation"], "language_creators": ["found"], "language": ["en", "nb", false, "nn", "sv", "da", "is", "fo"], "license": ["other"], "multilinguality": ["multilingual"], "size_categories": ["2G<n<1B"], "source_datasets": ["original"], "task_categories": ["text-generation"], "task_ids": ["language-modeling"], "pretty_name": "NCC", "extra_gated_prompt": "The Directive on Copyright in the Digital Single Market, which came into force on June 6 2019, amends the European Union copyright and database legislation and allows for Text and Data Mining (TDM) activities for research organizations and cultural heritage institutions. Under the terms of the aforementioned directive, by clicking on 'Access repository' you agree on using the text and data contained in this dataset for non-commercial scientific purposes only."}
TAGS
#task_categories-text-generation #task_ids-language-modeling #annotations_creators-no-annotation #language_creators-found #multilinguality-multilingual #size_categories-2G<n<1B #source_datasets-original #language-English #language-Norwegian Bokmål #language-Norwegian #language-Norwegian Nynorsk #language-Swedish #language-Danish #language-Icelandic #language-Faroese #license-other #arxiv-2104.09617 #region-us 

Dataset Card for NbAiLab/NCC
============================


Table of Contents
-----------------


* Dataset Description
* Dataset Summary
* Data Fields
* Dataset Creation
* Statistics
* Document Types
* Languages
* Publish Periode
* Considerations for Using the Data
* Social Impact of Dataset
* Discussion of Biases
* Other Known Limitations
* Additional Information
* Dataset Curators
* Licensing Information
* Citation Information


Dataset Description
-------------------


* Homepage: URL
* Repository: URL
* Paper: URL
* Point of Contact: Freddy Wetjen


The Norwegian Colossal Corpus is a collection of multiple smaller Norwegian corpuses suitable for training large language models. We have done extensive cleaning on the datasets, and have made them available in a common format. The total size of the NCC is currently 45GB.


How to Use
----------


Download Data
-------------


If you do not want to use the HuggingFace Dataset-library for training, or if you want to do additional pre-processing, it is also possible to download the files locally.



List of all the files.
* train-shard-0001-of-0046
* train-shard-0002-of-0046
* train-shard-0003-of-0046
* train-shard-0004-of-0046
* train-shard-0005-of-0046
* train-shard-0006-of-0046
* train-shard-0007-of-0046
* train-shard-0008-of-0046
* train-shard-0009-of-0046
* train-shard-0010-of-0046
* train-shard-0011-of-0046
* train-shard-0012-of-0046
* train-shard-0013-of-0046
* train-shard-0014-of-0046
* train-shard-0015-of-0046
* train-shard-0016-of-0046
* train-shard-0017-of-0046
* train-shard-0018-of-0046
* train-shard-0019-of-0046
* train-shard-0020-of-0046
* train-shard-0021-of-0046
* train-shard-0022-of-0046
* train-shard-0023-of-0046
* train-shard-0024-of-0046
* train-shard-0025-of-0046
* train-shard-0026-of-0046
* train-shard-0027-of-0046
* train-shard-0028-of-0046
* train-shard-0029-of-0046
* train-shard-0030-of-0046
* train-shard-0031-of-0046
* train-shard-0032-of-0046
* train-shard-0033-of-0046
* train-shard-0034-of-0046
* train-shard-0035-of-0046
* train-shard-0036-of-0046
* train-shard-0037-of-0046
* train-shard-0038-of-0046
* train-shard-0039-of-0046
* train-shard-0040-of-0046
* train-shard-0041-of-0046
* train-shard-0042-of-0046
* train-shard-0043-of-0046
* train-shard-0044-of-0046
* train-shard-0045-of-0046
* train-shard-0046-of-0046
* validation-shard-0001-of-0001



### Dataset Summary


The NCC dataset contains json lines with language training data. Here is an example json line:


Data Fields
-----------



### Dataset Creation


We are providing a train and a validation split. The standard size of the validation is a single 1GB file, while train is sharded in 1GB chunks.
All files are gzipped.


Build date: 21012022


#### Initial Data Collection and Curation


The procedure for the dataset creation is described in detail in our paper.


### Summary



### Document Types



### Languages



### Publish Periode



Considerations for Using the Data
---------------------------------


This corpus contains data under copyright and is not allowed to be used outide the National Library of Norway. The dataset should not be distributed.


### Discussion of Biases


Please refer to our paper.


### Dataset Curators


Freddy Wetjen and Per Egil Kummervold


License
-------


Various licences applies to different parts of the corpus. Every document in the corpus has a tag telling what "doc\_type" it belongs to. If you are unable to accept any of the licenses, you should filter out the "doc\_type" with a conflicting license.



We are preparing an article with detailed information about this corpus. Until it is published, please cite out paper discussing the first version of this corpus:



### Dataset Summary


The NCC dataset contains json lines with language training data. Here is an example json line:


Data Fields
----------- ### Dataset Creation


We are providing a train and a validation split. The standard size of the validation is a single 1GB file, while train is sharded in 1GB chunks.
All files are gzipped.


Build date: 21012022 #### Initial Data Collection and Curation


The procedure for the dataset creation is described in detail in our paper. ### Summary ### Document Types ### Languages ### Publish Periode



Considerations for Using the Data
---------------------------------


This corpus contains data under copyright and is not allowed to be used outide the National Library of Norway. The dataset should not be distributed. ### Discussion of Biases


Please refer to our paper. ### Dataset Curators


Freddy Wetjen and Per Egil Kummervold


License
-------


Various licences applies to different parts of the corpus. Every document in the corpus has a tag telling what "doc\_type" it belongs to. If you are unable to accept any of the licenses, you should filter out the "doc\_type" with a conflicting license.



We are preparing an article with detailed information about this corpus. Until it is published, please cite out paper discussing the first version of this corpus:
TAGS
#task_categories-text-generation #task_ids-language-modeling #annotations_creators-no-annotation #language_creators-found #multilinguality-multilingual #size_categories-2G<n<1B #source_datasets-original #language-English #language-Norwegian Bokmål #language-Norwegian #language-Norwegian Nynorsk #language-Swedish #language-Danish #language-Icelandic #language-Faroese #license-other #arxiv-2104.09617 #region-us 
 ### Dataset Summary


The NCC dataset contains json lines with language training data. Here is an example json line:


Data Fields
----------- ### Dataset Creation


We are providing a train and a validation split. The standard size of the validation is a single 1GB file, while train is sharded in 1GB chunks.
All files are gzipped.


Build date: 21012022 #### Initial Data Collection and Curation


The procedure for the dataset creation is described in detail in our paper. ### Summary ### Document Types ### Languages ### Publish Periode



Considerations for Using the Data
---------------------------------


This corpus contains data under copyright and is not allowed to be used outide the National Library of Norway. The dataset should not be distributed. ### Discussion of Biases


Please refer to our paper. ### Dataset Curators


Freddy Wetjen and Per Egil Kummervold


License
-------


Various licences applies to different parts of the corpus. Every document in the corpus has a tag telling what "doc\_type" it belongs to. If you are unable to accept any of the licenses, you should filter out the "doc\_type" with a conflicting license.



We are preparing an article with detailed information about this corpus. Until it is published, please cite out paper discussing the first version of this corpus:
passage: TAGS
#task_categories-text-generation #task_ids-language-modeling #annotations_creators-no-annotation #language_creators-found #multilinguality-multilingual #size_categories-2G<n<1B #source_datasets-original #language-English #language-Norwegian Bokmål #language-Norwegian #language-Norwegian Nynorsk #language-Swedish #language-Danish #language-Icelandic #language-Faroese #license-other #arxiv-2104.09617 #region-us 
### Dataset Summary


The NCC dataset contains json lines with language training data. Here is an example json line:


Data Fields
-----------### Dataset Creation


We are providing a train and a validation split. The standard size of the validation is a single 1GB file, while train is sharded in 1GB chunks.
All files are gzipped.


Build date: 21012022#### Initial Data Collection and Curation


The procedure for the dataset creation is described in detail in our paper.### Summary### Document Types### Languages### Publish Periode



Considerations for Using the Data
---------------------------------


This corpus contains data under copyright and is not allowed to be used outide the National Library of Norway. The dataset should not be distributed.### Discussion of Biases


Please refer to our paper.### Dataset Curators


Freddy Wetjen and Per Egil Kummervold


License
-------


Various licences applies to different parts of the corpus. Every document in the corpus has a tag telling what "doc\_type" it belongs to. If you are unable to accept any of the licenses, you should filter out the "doc\_type" with a conflicting license.



We are preparing an article with detailed information about this corpus. Until it is published, please cite out paper discussing the first version of this corpus: