
# Latvian text dataset

Data set of latvian language texts. Intended for use in AI tool development, like speech recognition or spellcheckers


## Data sources used
* Latvian Wikisource articles - https://wikisource.org/wiki/Category:Latvian
* Literary works of Rainis - https://repository.clarin.lv/repository/xmlui/handle/20.500.12574/41
* Latvian Wikipedia articles - https://huggingface.co/datasets/joelito/EU_Wikipedias
* European Parliament Proceedings Parallel Corpus - https://huggingface.co/datasets/europarl_bilingual
* Tilde MODEL Corpus - Multilingual Open Data for European Languages - https://huggingface.co/datasets/tilde_model



To get Wikipedia dataset (197MB) run. 
```
python tools/wikipedia/GetWikipedia.py
```

To get Europarl dataset (1.7GB) run. 
```
python tools/europarl/GetEuroparl.py
```

To get Tilde dataset (834MB) run. 
```
python tools/europarl/GetTilde.py
```

To combine all datasets run 
```
sh combine-all.sh
```

To clean out some junk run.
```
sh clean.sh
```

Also maybe you want to remove duplocate lines. To do so run 
```
sort lv.txt | uniq > lv-uniq.txt
```

## Notes

Possible future sources
* Parliament proceedings transcripts - https://www.saeima.lv/lv/transcripts
* Discussions of Latvian Wikipedia pages - https://lv.wikipedia.org/wiki/Special:AllPages
* Out of copyright books from LNB collection - https://data.gov.lv/dati/lv/dataset/gramatu-digitala-kolekcija

Data sets not used
* Web scrapes, as they tend to yield data from comments with improper spelling like "atrashanaas vieta" instead of "atrašanās vieta"
* Open Subtitles, as they contain data with improper spelling like "atrashanaas vieta" instead of "atrašanās vieta"

Possible issues:
* Data sets contain foreign language characters, like "蠻子" or cyrilic f.e. "Рига"
task_categories:automatic-speech-recognition annotations_creators:found language_creators:found multilinguality:monolingual size_categories:10K<n<100K source_datasets:extended|tilde_model source_datasets:extended|wikipedia source_datasets:extended|europarl_bilingual language:lv license:cc-by-4.0 lv latvian region:us
{"annotations_creators": ["found"], "language_creators": ["found"], "language": ["lv"], "license": ["cc-by-4.0"], "multilinguality": ["monolingual"], "size_categories": ["10K<n<100K"], "source_datasets": ["extended|tilde_model", "extended|wikipedia", "extended|europarl_bilingual"], "task_categories": ["automatic-speech-recognition"], "task_ids": [], "pretty_name": "Latvian text dataset", "tags": ["lv", "latvian"]}
TAGS
#task_categories-automatic-speech-recognition #annotations_creators-found #language_creators-found #multilinguality-monolingual #size_categories-10K<n<100K #source_datasets-extended|tilde_model #source_datasets-extended|wikipedia #source_datasets-extended|europarl_bilingual #language-Latvian #license-cc-by-4.0 #lv #latvian #region-us 


# Latvian text dataset

Data set of latvian language texts. Intended for use in AI tool development, like speech recognition or spellcheckers


## Data sources used
* Latvian Wikisource articles - URL
* Literary works of Rainis - URL
* Latvian Wikipedia articles - URL
* European Parliament Proceedings Parallel Corpus - URL
* Tilde MODEL Corpus - Multilingual Open Data for European Languages - URL



To get Wikipedia dataset (197MB) run. 


To get Europarl dataset (1.7GB) run. 


To get Tilde dataset (834MB) run. 


To combine all datasets run 


To clean out some junk run.


Also maybe you want to remove duplocate lines. To do so run 


## Notes

Possible future sources
* Parliament proceedings transcripts - URL
* Discussions of Latvian Wikipedia pages - URL
* Out of copyright books from LNB collection - URL

Data sets not used
* Web scrapes, as they tend to yield data from comments with improper spelling like "atrashanaas vieta" instead of "atrašanās vieta"
* Open Subtitles, as they contain data with improper spelling like "atrashanaas vieta" instead of "atrašanās vieta"

Possible issues:
* Data sets contain foreign language characters, like "蠻子" or cyrilic f.e. "Рига"
# Latvian text dataset

Data set of latvian language texts. Intended for use in AI tool development, like speech recognition or spellcheckers ## Data sources used
* Latvian Wikisource articles - URL
* Literary works of Rainis - URL
* Latvian Wikipedia articles - URL
* European Parliament Proceedings Parallel Corpus - URL
* Tilde MODEL Corpus - Multilingual Open Data for European Languages - URL



To get Wikipedia dataset (197MB) run. 


To get Europarl dataset (1.7GB) run. 


To get Tilde dataset (834MB) run. 


To combine all datasets run 


To clean out some junk run.


Also maybe you want to remove duplocate lines. To do so run ## Notes

Possible future sources
* Parliament proceedings transcripts - URL
* Discussions of Latvian Wikipedia pages - URL
* Out of copyright books from LNB collection - URL

Data sets not used
* Web scrapes, as they tend to yield data from comments with improper spelling like "atrashanaas vieta" instead of "atrašanās vieta"
* Open Subtitles, as they contain data with improper spelling like "atrashanaas vieta" instead of "atrašanās vieta"

Possible issues:
* Data sets contain foreign language characters, like "蠻子" or cyrilic f.e. "Рига"
TAGS
#task_categories-automatic-speech-recognition #annotations_creators-found #language_creators-found #multilinguality-monolingual #size_categories-10K<n<100K #source_datasets-extended|tilde_model #source_datasets-extended|wikipedia #source_datasets-extended|europarl_bilingual #language-Latvian #license-cc-by-4.0 #lv #latvian #region-us 
 # Latvian text dataset

Data set of latvian language texts. Intended for use in AI tool development, like speech recognition or spellcheckers ## Data sources used
* Latvian Wikisource articles - URL
* Literary works of Rainis - URL
* Latvian Wikipedia articles - URL
* European Parliament Proceedings Parallel Corpus - URL
* Tilde MODEL Corpus - Multilingual Open Data for European Languages - URL



To get Wikipedia dataset (197MB) run. 


To get Europarl dataset (1.7GB) run. 


To get Tilde dataset (834MB) run. 


To combine all datasets run 


To clean out some junk run.


Also maybe you want to remove duplocate lines. To do so run ## Notes

Possible future sources
* Parliament proceedings transcripts - URL
* Discussions of Latvian Wikipedia pages - URL
* Out of copyright books from LNB collection - URL

Data sets not used
* Web scrapes, as they tend to yield data from comments with improper spelling like "atrashanaas vieta" instead of "atrašanās vieta"
* Open Subtitles, as they contain data with improper spelling like "atrashanaas vieta" instead of "atrašanās vieta"

Possible issues:
* Data sets contain foreign language characters, like "蠻子" or cyrilic f.e. "Рига"