---
title             : "grafzahl: fine-tuning Transformers for text data from within R"
shorttitle        : "PUT THE R BACK IN TRANSFORMERS"

author: 
  - name          : "Chung-hong Chan"
    affiliation   : "1"
    corresponding : yes
    address       : "Unter Sachsenhausen 6-8, 50667 Köln"
    email         : "chainsawtiney@gmail.com"

affiliation:
  - id            : "1"
    institution   : "GESIS - Leibniz-Institut für Sozialwissenschaften, Germany"
    
authornote: |
  Source code and data are available at (redacted). The author(s) would like to thank Jarvis Labs for providing discounted GPU cloud service for the development of this package.

abstract: |
  This paper introduces `grafzahl`, an R package for fine-tuning Transformers for text data from within R. The package combines the ease of use of the `quanteda` R ecosystem and the state-of-the-art `Transformers` Python library. The package is used in this paper to reproduce the analyses in communication papers or, of non-Germanic benchmark datasets. Very significant improvement in model accuacy over traditional machine learning approach such as Convoluted Neural Network is observed. `grafzahl` might have a role in the mainstreamization of Transformer-based machine learning methods for communication research and beyond.
keywords          : "machine learning, transformers, R, python, automated content analysis"
wordcount         : "5649"

bibliography      : ["/home/chainsawriot/dev/dotfiles/bib.bib"]

floatsintext      : yes
figurelist        : no
tablelist         : no
figsintext        : yes
footnotelist      : no
linenumbers       : no
mask              : yes
draft             : no

documentclass     : "apa6"
classoption       : "man"
output:
 papaja::apa6_pdf:
   latex_engine: xelatex
---

```{r setup, include = FALSE}
library("papaja")
```

```{r analysis-preferences}
# Seed for random number generation
set.seed(42)
knitr::opts_chunk$set(cache.extra = knitr::rand_seed)
```

# The language for training supervised machine learning models

*"If supervised machine learning is at the core of your project, it may save you a lot of cursing to do this in Python."* [@van2022computational]

When it comes to comparing R and Python, there is an impression that R is not suitable for supervised machine learning. The above quote from a recent textbook on computational communication research captures that sentiment.

Undoubtedly, there are many great Python libraries available that makes Python **the** language for training supervised machine learning models. Some examples of these libraries are `scikit-learn` [@pedregosa2011scikit], `keras` [@chollet2015keras], and `transformers` [@wolf-etal-2020-transformers]. The last example is particular important because pre-trained Transformers (sometimes called Large Language Models or Foundation Models) such as BERT [@devlin2018bert] and the associated technique of fine-tuning [@howard:2018:ULM] (also known as transfer learning) have been extremely powerful for social scientific applications [^FINE]. In one such social scientific application [@widmann:2022:CCD], for example, a Transformer-based model provides in average 18-point higher F1 scores than dictionary-based methods; 9-point higher than the so-called end-to-end supervised neural network model with locally trained word embeddings (loosely dubbed as one of the "standard deep learning architectures").

[^FINE]: A pretrained neural network model on a certain task can capture the general knowledge about languages. The idea of fine-tuning is to freeze certain layers in a pretrained neural network, thereby keeps the general knowledge about languages. And then remove the last few layers (or even remove just the output layer), and retrain (or fine-tune) the last few layers of the model using the new data. The advantage of this technique is that the model doesn't need to learn the general knowledge again from the training data, but yet the model only needs to learn the domain knowledge from the data.

## Put the R back in Transformers

Many communication researchers were (and still are) introduced to text analysis with R [@rcore]. In 2017, a co-author of the aforementioned textbook wrote: "R is a powerful platform for computational text analysis, that can be a valuable tool for communication research. ... its well developed packages provide easy access to cutting edge text analysis techniques." [@welbers:2017:TAR, p. 262] `quanteda` [@benoit:2018], probably the most used R package by communication researchers for text analysis, has been extended with packages such as `newsmap` [@watanabe:2017:N], `LSX` [@watanabe2021latent], and `rectr` [@chan:2020:REC]. The papers of these packages were published in communication journals, a telling sign of their usefulness for communication research.

As many supervised machine learning tools are exclusively in Python, communication researchers who use R need to make a choice: 1) export the data from R to Python and then do the analysis there; or 2) switch the workflow entirely to Python. The author(s) of this paper saw both patterns [^PATTERNS].

[^PATTERNS]: An Example of the former: https://github.com/vanatteveldt/ecosent/ and the latter: https://github.com/annekroon/dictionaries-vs-sml

The purpose of this R package, `grafzahl`, is to provide the missing link between the `quanteda` ecosystem and modern Transformers language models. Under the hood, the training part is based on the Python packages `transformers` [@wolf-etal-2020-transformers] and `simpletransformers` [@simpletransformers]. The integration based on `reticulate` [@reticulate] is seamless. With this seamless integration provided, communication researchers can produce the most advanced supervised learning models entirely from within R. Therefore, there is no need to switch back and forth between programming languages. This package provides the function `grazahl()`, which emulates the behaviors of `quanteda.textmodels` [@quantedatextmodels]. Thus, users of that package would find the current package extremely familiar. The author(s) of this package argue that there is almost zero overhead for `quanteda` users to use the current package [^quanteda].

[^quanteda]: For R users who are not familiar with `quanteda`, the official website of the package (https://quanteda.io/) provides excellent tutorials. The end of this paper also provides an express introduction to the `corpus` object.

This package uses reasonable default settings which suit what communication researchers would like to achieve with these pretrained Transformers models. But the package also provides the freedom for communication researchers to finely adjust the parameters for their specific applications. However, the reanalysis of several examples in communication suggests that even the default settings can generate great improvement over the performance as reported in the original papers. Also, there is almost no need to conduct the cumbersome proprocessing and feature engineering steps, which all examples originally required.

An installable, anonymized version of the package is available from this view-only link: https://osf.io/dfhe5/?view_only=c9fbae773c7e4732ba7e638d08cedc02

See Appendix III for the installation guide.

## Related work

As of writing, `grafzahl` is the only R package that makes Transformer-based supervised learning easier. There are several similar packages. `text` [@kjell_giorgi_schwartz_2021] and `golgotha` [@golgotha] are two packages which also link the Python package `transformers` to R. But the two packages focus on generating contextual embeddings, not as an all-in-one "two-line" solution for supervised machine learning. See the examples below for the "two-line" solution.

A blog post by RStudio AI [@abdullayev2020state-of-the-art] explains how to integrate the Python package `transformers` from scratch using `reticulate`. The method involves using `transformers` to preprocess the text data and then `keras` for supervised machine learning.

For Python users, there have been several easy-to-use solutions for supervised learning with Transformers. `simpletransformers`, which `grafzahl` also links to, and `happytransformer` [^happytransformer] are two examples.

[^happytransformer]: https://happytransformer.com/

# Default workflow

The default settings of this package assume a usual "early stopping" workflow as following:

1. The labeled data have been split into two sets: training set and test set. This step must be done manually because it is not entirely a methodological issue.
2. The input data to the training function is the training set
3. The training set is further split in a 80:20 basis into the actual training set (ATS) and cross-validation set (CVS).
4. In an epoch, ATS is used to fine-tune a Transformer. CVS is used to evaluate the out-of-sample performance.
5. If the out-of-sample performance does not improve for two epochs, the training stops.

It is possible to override this default by manually adjusting the parameter `train_size` to exactly 1 and control the number of epochs with `num_train_epochs`.

# Monolingual classification example

@atteveldt:2021:VSA compare various methods to analyze the tone of Dutch economic news' headlines. Headlines were coded into three categories: negative (-1), neutral (0), and positive (+1).

In the original paper, @atteveldt:2021:VSA show that the best method for predicting expert coding, other than coding by student helpers, is convoluted neural network (CNN) with Dutch word embeddings trained on Dutch news. The out-of-sample F1 of .63, .66, and .56 were reported for the three categories. As the data (including the training-and-test split) are publicly available [^wouter], the author(s) of this paper can provide a head-to-head comparison between the reported CNN and the Transformer-based model trained with the current package.

[^wouter]: https://github.com/vanatteveldt/ecosent/

In the released data (`sentences_ml.csv`), there are three important columns:

1. `headline`: the actual text data
2. `value`: the sentiment
3. `gold`: whether or not this row is "gold standard", i.e. test set. There are 6,038 and 300 headlines in the training and test set respectively. 

## Workflow

### Step 0: Setup `grafzahl`

This step only needs to be done once. A miniconda environment needs to be setup. If there is a GPU capable of performing CUDA, run:

```r
require(grafzahl)
setup_grafzahl(cuda = TRUE) # set to FALSE otherwise
```

### Step 1: Get information of the pretrained Transformer

The first step of training a Transformer-based model is to find a suitable pretrained Transformer model on Hugging Face [^hugg], which would work for the data. As the data are in Dutch, a pretrained Dutch Transformer model such as BERTje should work [@de2019bertje, available from https://huggingface.co/GroNLP/bert-base-dutch-cased]. The model name of this model is `GroNLP/bert-base-dutch-cased`. It is also important to note the citation information to properly cite the pretrained Transformer model.

[^hugg]: Hugging Face (https://huggingface.co) is an online repository of pretrained machine learning models. 

### Step 2: Create the corpus

The second step is to read the data as a corpus [^CORPUS]. The easiest way to do that is to use the R package `readtext` [@readtext].

[^CORPUS]: This step is not absolutely needed. The package can also work with character vectors. The `corpus` data structure, which is a better representation of character vector, makes the workflow easier. See Appendix II for different ways to use the `grafzahl()` function.

```r
require(readtext)
require(quanteda)
input <- readtext("sentences_ml.csv", text_field = "headline") %>%
    corpus
```

We can manipulate the corpus object using the functions provided by `quanteda`. For example, one can subset the training set using the function `corpus_subset()`.

```r
## selecting documents where the docvar `gold` is FALSE
training_corpus <- corpus_subset(input, !gold)
```

### Step 3: Fine-tune the model

With the corpus and model name, the `grafzahl` function is used to fine-tune the model.

```r
model <- grafzahl(x = training_corpus,
                  y = "value",
                  model_name = "GroNLP/bert-base-dutch-cased")
```

In general, it is better to specify `output_dir` (where to put the saved model object). By default, it will be `output` in the current working directory. The R function `set.seed()` can also be used to preserve the random seed for reproducibility.

On a regular off-the-shelf gaming laptop with a GeForce RTX 3050 Ti GPU and 4G of GPU ram, the process took around 20 minutes. It is in general not recommended to train this without a CUDA-compatible GPU. In those cases, the process might take days, if not weeks.

### Step 4: Make prediction

Following the convention of `lm()` and many other R packages, the object returned by the function `grafzahl()` has a `predict()` S3 method. The following code gets the predicted sentiment of the headlines in the test set.

```r
test_corpus <- corpus_subset(input, gold)
predicted_sentiment <- predict(model, test_corpus)
```

### Step 5: Evaluate performance

With the predicted sentiment and the ground truth, there are many ways to evaluate the performance of the fine-tuned model. The simplest way is to construct a confusion matrix using the standard `table()` function.

```r
cm <- table(predicted_sentiment, ground_truth = docvars(test_corpus, "value"))
```

The R package `caret` can also be used to calculate standard performance metrics such as Precision, Recall, and F1 [^caret].

```r
require(caret)
confusionMatrix(cm, mode = "prec_recall")
```

The out-of-sample F1 measures of the fine-tuned model are .76, .67, and .72 (vs reported .63, .66, and .56). There is great improvement over the CNN model reported by @atteveldt:2021:VSA, although the prediction accuracy for the neutral category is just on par. The overall accuracy of the fine-tuned model is similar to the case of "Crowd-coding (Single Coder)" reported in the original paper.

[^caret]: The function `confusionMatrix()` can accept the predicted values and ground truth directly, without using `table()` first. But the predicted values and ground truth must be `factor`: `confusionMatrix(as.factor(predicted_sentiment), as.factor(docvars(test_corpus, "value")), mode = "prec_recall")`. 

### Step 5: Explain the prediction

Unlike "glass-box" machine learning models [@dobbrick:2021:ETI], Transformer-based prediction models are "black-box". There are so many parameters in Transformers (the BERT base model has 110 million parameters) and this complexity makes each individual parameter of a model not interpretable.

A reasonable compromise is to make the prediction *explainable* instead. Generating Local Interpretable Model-agnostic Explanations (LIME) [@ribeiro2016should; R implementation by @lime] is a good way to explain how the model makes its prediction. The gist of the method is to perturb the input text data by deleting parts of the sentence. For example: the sentence "I hate this movie" will be perturbed as "I this movie", "I hate movie", "I hate this", "I hate" etc. These perturbed sentences are then feed into the machine learning model to make predictions. The relationship between what get deleted and the prediction is studied. The parts that change the prediction a lot would be more *causal* to the original prediction.

With the trained model, we can explain the predictions made for the following two Dutch headlines: *"Dijsselbloem pessimistisch over snelle stappen Grieken"* (Dijsselbloem [the Former Minister of Finance of the Netherlands] pessimistic about rapid maneuvers from Greeks) and *"Aandelenbeurzen zetten koersopmars voort"* (Stock markets continue to rise). Models trained with `grafzahl` support the R package `lime` directly. One can get explanations using the following code:

```r
require(lime)
sentences <- c("Dijsselbloem pessimistisch over snelle stappen Grieken",
               "Aandelenbeurzen zetten koersopmars voort")
explainer <- lime(training_corpus, model)
explanations <- explain(sentences, explainer, n_labels = 1,
                        n_features = 3)
plot_text_explanations(explanations)
```

```{r fig1, echo = FALSE, fig.cap = 'Generating Local Interpretable Model-agnostic Explanations (LIME) of two predictions from the trained Dutch sentiment model', out.width = "100%"}
knitr::include_graphics("fig1.png")
```

Figure \@ref(fig:fig1) shows that for the sentence *"Dijsselbloem pessimistisch over snelle stappen Grieken"* (classified as negative), the tokens *pessimistisch* and *stappen* are making the prediction towards the classified position (negative). But the token *Dijsselbloem* is making it away.

# Multilingual regression example

@dobbrick:2021:ETI present a study of comparing various methods to learn and predict integrative complexity of English and German online user comments from Facebook, Twitter, and news website comment sections. According to the original paper, "Integrative complexity is a psychological measure that researchers increasingly implement to assess the argumentative quality of public debate contributions." [@dobbrick:2021:ETI, p. 3] Comments were coded with a standard coding scheme into a 7-point Likert scale from 1 (lowest complexity) to 7 (highest complexity). The paper presents two approaches: Assumption-based approach and Shotgun approach. The Shotgun approach is similar to the traditional full-text machine learning approach used in the previous example [@atteveldt:2021:VSA]. Similar to @atteveldt:2021:VSA, @dobbrick:2021:ETI report that CNN with word embeddings provides the best out-of-sample performance. The original paper reports 10-fold cross-validation. Root mean squared error (RMSE) of .75 (English) and .84 (German) were reported. It is also important to note that @dobbrick:2021:ETI trained an individual model for each language. Similar to @atteveldt:2021:VSA, the human annotated data and the original training-and-test split are publicly available [^JULIA]. In total, there are 4,800 annotated comments.

[^JULIA]: https://osf.io/578mg/

## Workflow

The workflow is similar to the one described above. One important modification is that the parameter `regression` is `TRUE` (default to `FALSE`) this time. Also, unlike the original paper [@dobbrick:2021:ETI], only one model based on the distilled multilingual BERT [@sanh2019distilbert] is trained. The distilled multilingual BERT supports 104 languages, therein include German and English. The distilled multilingual BERT retains 97\% of the performance of the original multilingual BERT but is 40\% smaller. Therefore, it is less computational expensive to fine-tune, especially useful in the case of fine-tuning on an off-the-shelf gaming laptop with a limited amount of GPU ram.

```r
mod <- grafzahl(x = en_corpus + de_corpus,
                y = "icom"
                model_name = "distilbert-base-multilingual-cased",
                regression = TRUE)
```

In the above code, it also highlights one benefit of working with `corpus`: `corpus` is addible.

## Results

Apply the same 10-fold cross-validation setup, the RMSE for English and German are .67 and .74 respectively (vs. .75 and .84 from the original paper, lower is better).

# Non-Germanic example: Amharic

The two examples above feature three Germanic languages: Dutch, English and German. They demonstrate remarkable performance of `grafzahl` in working on Germanic languages, even cross-lingually. However, the author(s) want to emphasize that `grafzahl` is not just another package focusing only on English, or Germanic languages in general. @baden:2021:TGC criticize this tendency: *"Researchers are far more likely to find and apply suitable CTAM (computational text analysis methods) for studying English texts, privileging anglophone researchers and anglophone research sites. Where researchers intend to study textual material in other languages, by contrast, CTAM often require considerable efforts at adaptation, fall short in performance or are entirely unavailable, with painful consequences for the diversification and de-Westernization of social science research"* (p. 11). 

In view of this, the current author(s) present two additional examples on how `grafzahl` can help to ease this English and Germanic languages dominance.

Amharic is a Semitic language mainly spoken in Ethiopia. After Arabic, Amharic is the second most-spoken Semitic language. Unlike many Semitic languages using the *abjad* (consonant-only) writing system, Amharic is written in a unique alphasyllabary writing system called *Ge'ez*. Syntactically, Amharic is also different from many Germanic languages for its SOV (subject-object-verb) word order [^SOV]. It is in general considered to be a "low resource" language. On @joshi2020state 's rating of 0 ("The Left-Behinds", such as the Polynesian language Wallisian) to 5 ("The Winners", such as English and German) on resourcefulness, Amharic has a rating of 2. Only recently, the first news classification dataset called "Amharic News Text classification Dataset" is available [@azime2021amharic].

[^SOV]: Actually, majority of the languages are SOV, while SVO (many Germanic languages) are slightly less common.

Amharic News Text classification Dataset contains 50,706 news articles curated from various Amharic websites. The original paper reports the baseline out-of-sample accuracy of 62.2\% using Naive Bayes. The released data also contains the training-and-test split [^Amharic]. It is a much bigger dataset than the two previous examples (training set: 41,185 articles, test set: 10,287). News articles were annotated into the following categories (originally written in *Ge'ez*, transliterated to Latin characters here): *hāgeri āk’efi zēna* (national news), *mezinanya* (entertainment), *siporiti* (sport), *bīzinesi* (business), *‘alemi āk’efi zēna* (international news), and *poletīka* (politics).

[^Amharic]: https://huggingface.co/datasets/israel/Amharic-News-Text-classification-Dataset/tree/main

In this example, the AfriBERTa is used as the pretrained model [@ogueji2021small]. The AfriBERTa model was trained with a small corpus of 11 African languages. Similar to the first two examples, the default settings of `grafzahl` are used.

```r
input <- readtext::readtext("am_train.csv", text_field = "article") %>%
    corpus %>% corpus_subset(category != "")

model <- grafzahl(x = input,
                  y = "category",
                  model_name = "castorini/afriberta_base")
```


## Results

The final out-of-sample accuracy is 84.18\%, a solid improvement from the baseline of 62.2\%.

# Non-Germanic example: Turkish

Turkish is a language of the Turkic language family and has a rating of 4 [@joshi2020state]. Similar to Amharic, its word order is also SOV.

In this example, we focus on SemEval 2020 Task 12 Subtask A [@zampieri-etal-2020-semeval;@ccoltekin2020corpus]. In this subtask, Turkish tweets, 31,756 and 3,528 in the training and test sets respectively, were coded as "Offensive" or "Not Offensive". The state-of-the-art performance by the world's best NLP experts for this subtask is 82.58\% (Marco F1). Of course, it is quite impossible for this R package with default settings to obtain this performance. But it would be interesting to see how well the performance this package could get.

In this example, the BERTurk model by the *Bayerische Staatsbibliothek* is used [^BERTurk] as the pretrained model.

[^BERTurk]: https://huggingface.co/dbmdz/bert-base-turkish-cased

```r
input <- readtext::readtext("offenseval-tr-training-v1.tsv",
                            text_field = "tweet", quote = "") %>% corpus

model <- grafzahl(x = input,
                  y = "subtask_a",
                  model_name = "dbmdz/bert-base-turkish-cased")
```

## Results

The final out-of-sample Macro F1 is 78.25\%, which is just 5\% shy of the state-of-the-art performance. If this package were released in 2020, this entry would have been the 8th best performance worldwide [@zampieri-etal-2020-semeval].


<!-- ## Korean -->

<!-- Korean is a language mainly spoken in the Korean peninsula and in it own language family (Koreanic). The example here is the Korean hate speech corpus [@moon2020beep]. The corpus composes of annotated entertainment news comments. This example is important because of the toxic fandom culture in South Korea [@lim2022south] and there have been several suicide cases of female artists attributed to the overtly sexist, negative online comments. -->

# Conclusion

This paper presents the R packages `grafzahl` and demonstrates its applicability to communication research by replicating the supervised machine learning part of published communication research. The author(s) of this paper believe that `grafzahl` could play an important role in mainstreamization of modern Transformer-based supervised machine learning. The "two-line solution", as illustrated in the various examples, shows how easy it is to produce models with almost the state-of-the-art performance straightly from the csv file.

The package inherits concerns from the whole notion of Transformer-based machine learning. The first concern is about the black-box nature of Transformers [@dobbrick:2021:ETI]. This black-box nature could also hide the social concerns of these complex systems such as social biases and potential harms [@boyarskaya2020overcoming]. The current author(s) acknowledge that Transformers are black-box. But the current R package is intended to help communication researchers to develop supervised machine learning models for automated content analyses. The potential harms, in comparison to other so-called AI systems, should be relatively small. But still, the current author(s) apply the Responsible AI Innovation framework by @boyarskaya2020overcoming to fully evaluate who is vulnerable. The first vulnerable group might be human coders, because their job might be at risk. Using the first example, @atteveldt:2021:VSA provide the learning curve of two machine learning algorithms: CNN and Support Vector Machine (SVM). A learning curve plots the out-of-sample prediction performance as a function of number of training examples. The current author(s) repeat the analysis in a similar manner to @atteveldt:2021:VSA and plot the learning curve of Transformer-based model trained using the default "early stopping" workflow of `grafzahl`. Figure \@ref(fig:fig2) show the fine-tuned Transformer model's learning curve alongside CNN's and SVM's. The fune-tuned model has much better performance than CNN and SVM even with only 500 training examples. Unlike CNN and SVM, the gain in performance appears to plateau after 2500. It points to the fact that one does not need to have a lot of training data to fine-tune a Transformer model. 

```{r fig2, fig.cap = "Learning curve of machine learning algorithms"}
readRDS(here::here("paper/learning.RDS"))
```

The second vulnerable group might be the actors who could be unfairly classified with negative attributes. For this group, the support for LIME [@ribeiro2016should] is at least helpful to diagnose the potential social biases. Figure \@ref(fig:fig1) shows how LIME could be useful for evaluating social biases. For the sentence, *"Dijsselbloem pessimistisch over snelle stappen Grieken"*, the LIME analysis shows that the features that lead to the negative evaluation are *pessimistisch* and *stappen*, but not the group name *Grieken*. The prediction model appears to be fair, at least for this prediction.

The second concern is the computational power required for this kind of model. Although no special hardware is required to run this software, CUDA-compatible GPUs could greatly speed up the process. Even the author(s) of `grafzahl` cannot afford to buy GPU(s) due to the current chip crunch (the main development was conducted on a rented gaming laptop). This might create another asymmetry to access and privilege those who could have access to these hardware, mainly those researchers from the rich Global North.

This cannot be solved by any technical solution and indeed GPU power is a valuable resource. The author(s) are working on making `grafzahl` runnable on the free tier of Google Colab and Amazon SageMaker Lab. This is not a solution but at least it helps to equalize the access.

The third concern is the carbon emission. The study by @strubell2019energy suggest that training BERT from scratch has the same carbon footprint to 5 cars in their lifetimes. The current author(s) argue that `grafzahl` is actually mitigating the problem by encouraging researchers to reuse (fine-tune) existing models and **not** to train any model from scratch. The carbon footprint to fine-tune a model is a tiny fraction of training BERT from scratch [^Estimation]. However, even with that, the procedure might still have a higher carbon footprint than other classic analytical techniques (classic machine learning such as regularized logistic regression or even dictionary-based method). Depending on tasks, Transformer-based supervised machine learning is not always helpful. In the first example, the neglible improvement of the Transformer-based model for classifying "neutral" Dutch headlines is the case in point.

[^Estimation]: Using the parameters from @strubell2019energy, the fine-tuning of the model in the first example emits .094 lbs of carbon(20 minutes of single GPU computing), or 0.00007% of **one** car's lifetime carbon emission.

The current author(s) ends this paper with the following: *"For unto whomsoever much is given, of him shall be much required: and to whom men have committed much, of him they will ask the more."* Please use this package cautiously and responsibly.

# Appendix I: Understanding `corpus` object

`quanteda` provides several data structures for working with text data, e.g. `corpus`, `tokens`, `dfm`, etc. For this paper, only the data structure `corpus` is important.

A `corpus` object holds documents. The simplest way to construct a `corpus` is from a character vector.

```{r, echo = TRUE}
library(quanteda)
x <- c("All human beings are born free and equal in dignity and rights.",
       "No one shall be subjected to arbitrary arrest, detention or exile.",
       "The history of all hitherto existing society is the history of
        class struggles.")
input <- corpus(x)
input
```

Another way to construct a `corpus` is to use the `readtext` package. Under the hood, a `corpus` object is still a character vector. What makes it different is the added metadata storage facilities.

The first level of metadata is corpus-level metadata. It can be manipulated using the `meta()` function.

```{r, echo = TRUE}
meta(input, "title") <- "Some famous sentences"
meta(input, "author") <- c("UN", "Karl Marx", "Friedrich Engels")
meta(input)
```

The second level of metadata is document-level metadata. In `quanteda`, document-level metadata are called `docvars`. Internally, `docvars` are stored as a `data.frame`.

```{r, echo = TRUE}
docvars(input)
```

A `docvar` can be entered and displayed this way:

```{r, echo = TRUE}
docvars(input, "source") <- c("UDHR", "UDHR", "The Communist Manifesto")
docvars(input)
```

```{r, echo = TRUE}
docvars(input, "source")
```

A better way to enter `docvars` is to use `readtext` (See the Dutch example above). All columns of the original file, except the `text_field`, are automatically entered as `docvars`.

The identifier of each document is called document names (`docnames`). The `corpus` function will create some default identifiers. One can manipulate `docnames` as below.

```{r, echo = TRUE}
docnames(input)
```

```{r, echo = TRUE}
docnames(input) <- c("UN1", "UN2", "CM1")
input
```

There are several functions for manipulating `corpus`. Like all `quanteda` functions which are named by the object to be manipulated, all functions for manipulating `corpus` are named `corpus_*()`. For example, the function `corpus_subset()` subsets a corpus based on a query of the `docvars`.

```{r, echo = TRUE}
corpus_subset(input, source != "UDHR")
```

## Interoperability

The `VCorpus` ("volatile corpus") object from the R package `tm` can be coerced as the `corpus` object.

```{r, echo = TRUE}
library(tm, warn.conflicts = FALSE)
tm_corpus <- VCorpus(VectorSource(x))
tm_corpus
```

```{r, echo = TRUE}
corpus(tm_corpus)
```

Unfortunately, tidytext objects are very specific to the "bag-of-words" approach of dealing with text. The tidy data frame objects cannot be converted to `corpus`.

# Appendix II: the `grafzahl()` function

## `x` and `y`

The `grafzahl()` is actually an S3 generic function. It supports both `corpus` and character vector, thereby makes the function very flexible.

Recalling the first example:

```r
require(readtext)
require(quanteda)
input <- readtext("sentences_ml.csv", text_field = "headline") %>%
    corpus

training_corpus <- corpus_subset(input, !gold)
model <- grafzahl(x = training_corpus,
                  y = "value",
                  model_name = "GroNLP/bert-base-dutch-cased")

```

If one doesn't want to create a `corpus`, another version of the same code is:

```r
input <- read.csv("sentences_ml.csv")
training_set <- input[!input$gold, ]

model <- grafzahl(x = training_set$headline,
                  y = training_set$value,
                  model_name = "GroNLP/bert-base-dutch-cased")
```

## `model_name` and `model_type`

The `model_name` parameter can either be the model name on Hugging Face (e.g. `xlm-roberta-base`, `GroNLP/bert-base-dutch-cased`, `distilbert-base-multilingual-cased`) or a local path.

As `model_name` can be a local path, it is possible to preserve a copy of the model locally for reproducibility purposes.

Suppose one wants to preserve the model `castorini/afriberta_base` locally. As all models on Hugging Face are stored as a Git repository, one can use git to clone the model locally. A cloned model usually takes around 1G of local storage.

```bash
## make sure you have installed git lfs
## https://git-lfs.github.com/
git lfs install
git clone https://huggingface.co/castorini/afriberta_base localafriberta
```

With the locally cloned model:

```r
input <- readtext::readtext("am_train.csv", text_field = "article") %>%
    corpus %>% corpus_subset(category != "")

model <- grafzahl(x = input,
                  y = "category",
                  model_name = "localafriberta")
```

In most of the situations, one doesn't need to care about `model_type`. All examples in this paper do not need to specify `model_type`, because `model_type` will be inferred automatically from `model_name` [^INFER].

[^INFER]: The mechanism is to analyze the `config.json` file (the so-called "model card"), either online on Hugging Face or in a local directory.

One important exception is `vinai/bertweet-base`. The model will be inferred as `roberta`. It is perfectly fine to use the inferred value. However, it would be better to provide the `model_type` manually as `bertweet` so that emojis are also included in the fine-tuning.

# Appendix III: Installation guide

Unfortunately, the R package `osfr` doesn't support view-only links. Therefore, one must download the file `grafzahl.tgz` manually.

```r
install.packages("grafzahl.tgz", dependencies = TRUE, repos = NULL)
```

# References

\begingroup
\setlength{\parindent}{-0.5in}
\setlength{\leftskip}{0.5in}

<div id="refs" custom-style="Bibliography"></div>
\endgroup
