
# Dataset Card for Faroese_BLARK_small

## Dataset Description

All sentences are retrieved from:

- **Paper:** 

Annika Simonsen, Sandra Saxov Lamhauge, Iben Nyholm Debess, and Peter Juel Henrichsen. 2022. Creating a Basic Language Resource Kit for Faroese. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4637–4643, Marseille, France. European Language Resources Association.

### Dataset Summary

This dataset is a filtered version of the corpus (35.6 M tokens) first published as BLARK - Basic Language Resource Kit for Faroese. 
The pre-processing and filtering steps include:

- Normalize format to utf-8
- Remove shorter sentences (less than 10 units, where units are separated by spaces)
- Remove archaic Faroese
- Remove separators ('\r', '\t', '\n')
- Remove non standard formatting. Examples: '§§', ' | ', '**', ' • ', '  •  ', '.- ', ': ?', '.?', '\xa0', '\xad', '_ _', '. .', etc.
- Remove (most) numbered lists, of formats: 1), 1:, Stk. 1 etc.
- Replace arbitrary number of question/exclamation marks and full-stops with 1. Example: !!!!!! -> !
- Remove websites that start with http
- Remove sentences without (or with little) linguistic content. In practice: all sentences where more than half of the characters (excluding spaces) are number, punctuations and letters in caps-lock (acronyms and initials)
- Remove duplicates

### Supported Tasks and Leaderboards

Suitable for MLM and CLM



task_categories:text-generation language:fo region:us
{"language": ["fo"], "task_categories": ["text-generation"]}
TAGS
#task_categories-text-generation #language-Faroese #region-us 


# Dataset Card for Faroese_BLARK_small

## Dataset Description

All sentences are retrieved from:

- Paper: 

Annika Simonsen, Sandra Saxov Lamhauge, Iben Nyholm Debess, and Peter Juel Henrichsen. 2022. Creating a Basic Language Resource Kit for Faroese. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4637–4643, Marseille, France. European Language Resources Association.

### Dataset Summary

This dataset is a filtered version of the corpus (35.6 M tokens) first published as BLARK - Basic Language Resource Kit for Faroese. 
The pre-processing and filtering steps include:

- Normalize format to utf-8
- Remove shorter sentences (less than 10 units, where units are separated by spaces)
- Remove archaic Faroese
- Remove separators ('\r', '\t', '\n')
- Remove non standard formatting. Examples: '§§', ' | ', '', ' • ', '  •  ', '.- ', ': ?', '.?', '\xa0', '\xad', '_ _', '. .', etc.
- Remove (most) numbered lists, of formats: 1), 1:, Stk. 1 etc.
- Replace arbitrary number of question/exclamation marks and full-stops with 1. Example: !!!!!! -> !
- Remove websites that start with http
- Remove sentences without (or with little) linguistic content. In practice: all sentences where more than half of the characters (excluding spaces) are number, punctuations and letters in caps-lock (acronyms and initials)
- Remove duplicates

### Supported Tasks and Leaderboards

Suitable for MLM and CLM



# Dataset Card for Faroese_BLARK_small ## Dataset Description

All sentences are retrieved from:

- Paper: 

Annika Simonsen, Sandra Saxov Lamhauge, Iben Nyholm Debess, and Peter Juel Henrichsen. 2022. Creating a Basic Language Resource Kit for Faroese. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4637–4643, Marseille, France. European Language Resources Association. ### Dataset Summary

This dataset is a filtered version of the corpus (35.6 M tokens) first published as BLARK - Basic Language Resource Kit for Faroese. 
The pre-processing and filtering steps include:

- Normalize format to utf-8
- Remove shorter sentences (less than 10 units, where units are separated by spaces)
- Remove archaic Faroese
- Remove separators ('\r', '\t', '\n')
- Remove non standard formatting. Examples: '§§', ' | ', '', ' • ', '  •  ', '.- ', ': ?', '.?', '\xa0', '\xad', '_ _', '. .', etc.
- Remove (most) numbered lists, of formats: 1), 1:, Stk. 1 etc.
- Replace arbitrary number of question/exclamation marks and full-stops with 1. Example: !!!!!! -> !
- Remove websites that start with http
- Remove sentences without (or with little) linguistic content. In practice: all sentences where more than half of the characters (excluding spaces) are number, punctuations and letters in caps-lock (acronyms and initials)
- Remove duplicates ### Supported Tasks and Leaderboards

Suitable for MLM and CLM
TAGS
#task_categories-text-generation #language-Faroese #region-us 
 # Dataset Card for Faroese_BLARK_small ## Dataset Description

All sentences are retrieved from:

- Paper: 

Annika Simonsen, Sandra Saxov Lamhauge, Iben Nyholm Debess, and Peter Juel Henrichsen. 2022. Creating a Basic Language Resource Kit for Faroese. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4637–4643, Marseille, France. European Language Resources Association. ### Dataset Summary

This dataset is a filtered version of the corpus (35.6 M tokens) first published as BLARK - Basic Language Resource Kit for Faroese. 
The pre-processing and filtering steps include:

- Normalize format to utf-8
- Remove shorter sentences (less than 10 units, where units are separated by spaces)
- Remove archaic Faroese
- Remove separators ('\r', '\t', '\n')
- Remove non standard formatting. Examples: '§§', ' | ', '', ' • ', '  •  ', '.- ', ': ?', '.?', '\xa0', '\xad', '_ _', '. .', etc.
- Remove (most) numbered lists, of formats: 1), 1:, Stk. 1 etc.
- Replace arbitrary number of question/exclamation marks and full-stops with 1. Example: !!!!!! -> !
- Remove websites that start with http
- Remove sentences without (or with little) linguistic content. In practice: all sentences where more than half of the characters (excluding spaces) are number, punctuations and letters in caps-lock (acronyms and initials)
- Remove duplicates ### Supported Tasks and Leaderboards

Suitable for MLM and CLM
passage: TAGS
#task_categories-text-generation #language-Faroese #region-us 
# Dataset Card for Faroese_BLARK_small## Dataset Description

All sentences are retrieved from:

- Paper: 

Annika Simonsen, Sandra Saxov Lamhauge, Iben Nyholm Debess, and Peter Juel Henrichsen. 2022. Creating a Basic Language Resource Kit for Faroese. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4637–4643, Marseille, France. European Language Resources Association.### Dataset Summary

This dataset is a filtered version of the corpus (35.6 M tokens) first published as BLARK - Basic Language Resource Kit for Faroese. 
The pre-processing and filtering steps include:

- Normalize format to utf-8
- Remove shorter sentences (less than 10 units, where units are separated by spaces)
- Remove archaic Faroese
- Remove separators ('\r', '\t', '\n')
- Remove non standard formatting. Examples: '§§', ' | ', '', ' • ', '  •  ', '.- ', ': ?', '.?', '\xa0', '\xad', '_ _', '. .', etc.
- Remove (most) numbered lists, of formats: 1), 1:, Stk. 1 etc.
- Replace arbitrary number of question/exclamation marks and full-stops with 1. Example: !!!!!! -> !
- Remove websites that start with http
- Remove sentences without (or with little) linguistic content. In practice: all sentences where more than half of the characters (excluding spaces) are number, punctuations and letters in caps-lock (acronyms and initials)
- Remove duplicates### Supported Tasks and Leaderboards

Suitable for MLM and CLM