@inproceedings{agirrezabal-etal-2023-hidden,
    title = "The Hidden Folk: Linguistic Properties Encoded in Multilingual Contextual Character Representations",
    author = "Agirrezabal, Manex  and
      Boldsen, Sidsel  and
      Hollenstein, Nora",
    editor = "Gorman, Kyle  and
      Sproat, Richard  and
      Roark, Brian",
    booktitle = "Proceedings of the Workshop on Computation and Written Language (CAWL 2023)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.cawl-1.2",
    doi = "10.18653/v1/2023.cawl-1.2",
    pages = "6--13",
    abstract = "To gain a better understanding of the linguistic information encoded in character-based language models, we probe the multilingual contextual CANINE model. We design a range of phonetic probing tasks in six Nordic languages, including Faroese as an additional zero-shot instance. We observe that some phonetic information is indeed encoded in the character representations, as consonants and vowels can be well distinguished using a linear classifier. Furthermore, results for the Danish and Norwegian language seem to be worse for the consonant/vowel distinction in comparison to other languages. The information encoded in these representations can also be learned in a zero-shot scenario, as Faroese shows a reasonably good performance in the same vowel/consonant distinction task.",
}

To gain a better understanding of the linguistic information encoded in character-based language models, we probe the multilingual contextual CANINE model. We design a range of phonetic probing tasks in six Nordic languages, including Faroese as an additional zero-shot instance. We observe that some phonetic information is indeed encoded in the character representations, as consonants and vowels can be well distinguished using a linear classifier. Furthermore, results for the Danish and Norwegian language seem to be worse for the consonant/vowel distinction in comparison to other languages. The information encoded in these representations can also be learned in a zero-shot scenario, as Faroese shows a reasonably good performance in the same vowel/consonant distinction task.
Agirrezabal, Manex Boldsen, Sidsel Hollenstein, Nora
The Hidden Folk: Linguistic Properties Encoded in Multilingual Contextual Character Representations