What methods have been proposed to categorize the kinds of editing operations that occur during text simplification?
Given a simplification dataset, automatic alignment
methods enable the extraction of atomic edits that
simplify the complex text. Prior work has analyzed
such edits to gain insights and compare datasets.
The most common analysis revolves around mea-
suring the frequency of different editing operations
(i.e. insertions, deletions, replacements) (Coster
and Kauchak, 2011; Vásquez-Rodríguez et al.,
2021). Some work has proposed annotating the
operations with linguistically motivated categories
that give a reason for the edit. Since most simpli-
fication resources are at the sentence granularity,
edit categorizations have focused on lexical and
syntactic phenomena that frequently occur within
individual sentences (Aluísio et al., 2008; Scarton
and Specia, 2018; Cardon et al., 2022)
Learning Simplifications for Specific Target Audiences
Text simplification (TS) is a monolingual text-to-text transformation task where an original (complex) text is transformed into a target (simpler) text. Most recent work is based on sequence-to-sequence neural models similar to those used for machine translation (MT). Different from MT, TS data comprises more elaborate transformations, such as sentence splitting. It can also contain multiple simplifications of the same original text targeting different audiences, such as school grade levels. We explore these two features of TS to build models tailored for specific grade levels. Our approach uses a standard sequence-to-sequence architecture where the original sequence is annotated with information about the target audience and/or the (predicted) type of simplification operation. We show that it outperforms state-of-the-art TS approaches (up to 3 and 12 BLEU and SARI points, respectively), including when training data for the specific complex-simple combination of grade levels is not available, i.e. zero-shot learning.
[{"first":"Carolina","last":"Scarton","middle":[]},{"first":"Lucia","last":"Specia","middle":[]}]
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
Evaluating automatic text simplification (ATS) systems is a difficult task that is either performed by automatic metrics or user-based evaluations. However, from a linguistic point-of-view, it is not always clear on what bases these evaluations operate. In this paper, we propose annotations of the ASSET corpus that can be used to shed more light on ATS evaluation. In addition to contributing with this resource, we show how it can be used to analyze SARI’s behavior and to re-evaluate existing ATS systems. We present our insights as a step to improve ATS evaluation protocols in the future.
[{"first":"Rémi","last":"Cardon","middle":[]},{"first":"Adrien","last":"Bibal","middle":[]},{"first":"Rodrigo","last":"Wilkens","middle":[]},{"first":"David","last":"Alfter","middle":[]},{"first":"Magali","last":"Norré","middle":[]},{"first":"Adeline","last":"Müller","middle":[]},{"first":"Watrin","last":"Patrick","middle":[]},{"first":"Thomas","last":"François","middle":[]}]
Simple English Wikipedia: A New Text Simpliﬁcation Task
In this paper we examine the task of sentence simpliﬁcation which aims to reduce the read-ing complexity of a sentence by incorporat-ing more accessible vocabulary and sentence structure. We introduce a new data set that pairs English Wikipedia with Simple English Wikipedia and is orders of magnitude larger than any previously examined for sentence simpliﬁcation. The data contains the full range of simpliﬁcation operations including rewording, reordering, insertion and deletion. We provide an analysis of this corpus as well as preliminary results using a phrase-based translation approach for simpliﬁcation.
[{"first":"William","last":"Coster","middle":[]},{"first":"David","last":"Kauchak","middle":[]}]
The Role of Text Simplification Operations in Evaluation
Research in Text Simplification (TS) has relied mostly on the Wikipedia-based datasets and the SARI evaluation metric, as the preferred means for creating and evaluating new simplification methods. Previous studies have pointed out the flaws of data evaluation resources, including incorrect alignment of simple/complex sentence pairs, sentences with no simplifications or a dearth in the variety of simplification operations. However, there are no further analyses on the impact of the original data distribution regarding the type of simplification operations performed. In this paper, we set up a systematic benchmark of the most common TS datasets, basing our evaluation on different protocols for split selection (e.g., selection by random or by Monte Carlo). We perform an operation-based investigation, demonstrating in detail the limitations of existing simplification datasets. Further, we make recommendations for future standardised practices in the design, creation and evaluation of TS resources.
[{"middle":[],"last":"Vásquez-Rodríguez","first":"Laura"},{"middle":[],"last":"Shardlow","first":"Matthew"},{"middle":[],"last":"Przybyla","first":"Piotr"},{"middle":[],"last":"Ananiadou","first":"Sophia"}]
Learning Simplifications for Specific Target Audiences
Text simplification (TS) is the task of modifying an original text into a simpler version of it. One of the main parameters for defining a suitable simplification is the target audience. Examples include elderly, children, cognitively impaired users, nonnative speakers and low-literacy readers.
Learning Simplifications for Specific Target Audiences
Traditionally, work on TS has been divided in lexical simplification (LS) and syntactic simplification (SS). LS (Paetzold, 2016) deals with the identification and replacement of complex words or phrases. SS (Siddharthan, 2011) performs structural transformations such as changing a sentence from passive to active voice. However, most recent approaches learn transformations from corpora, addressing simplification at lexical and syntactic levels altogether. These include either learning tree-based transformations (Woodsend and Lapata, 2011;Paetzold and Specia, 2013) or using machine translation (MT)-based techniques (Zhu et al., 2010;Coster and Kauchak, 2011a;Wubben et al., 2012;Narayan and Gardent, 2014;Nisioi et al., 2017;Zhang and Lapata, 2017). This paper uses the latter type of technique, which treats TS as a monolingual MT task, where an original text is "translated" into its simplified version.
Learning Simplifications for Specific Target Audiences
In order to build MT-based models, a parallel corpus of original texts with their simplified counterparts is needed. For English, two main such corpora are available: Wikipedia-Simple Wikipedia (W-SW) (Zhu et al., 2010) and the Newsela Article Corpus. 1 The former is a collection of original Wikipedia articles and their simplified versions created by volunteers. The latter consists of news articles professionally simplified for various specific audiences following the US school grade system. To build simplification models, the pairs of articles in these corpora have been aligned at the level of smaller units using standard algorithms (Coster and Kauchak, 2011b;Paetzold and Specia, 2016;Štajner et al., 2017). Based on the number of sentences involved in these alignments, one can categorise alignments into four types of coarse-grained simplification operations:
Learning Simplifications for Specific Target Audiences
• Identical: an original sentence is aligned to itself, i.e. no simplification is performed. • Elaboration: an original sentence is aligned to a single, rewritten simplified sentence. • One-to-many: splitting -an original sentence is aligned to 2+ simplified sentences.
Learning Simplifications for Specific Target Audiences
• Many-to-one: joining -2+ original sentences are aligned to a single simplified sentence.
Learning Simplifications for Specific Target Audiences
We hereafter refer to the unit of simplification, i.e. one or more original or simplified sentences, as instances.
Learning Simplifications for Specific Target Audiences
The Newsela corpus is seen as having higher quality than W-SW because its simplifications are created by professionals, following well defined guidelines (Xu et al., 2015). It is also larger which is preferable for training corpus-based models. More interestingly, the Newsela corpus has a feature that has been ignored thus far: Each instance in the corpus was created for readers with a certain school grade level. Each original article has a label indicating its corresponding grade level (from 12 to 2), and may have various simplified versions, each for a different grade level. For example, a level 12 article may have simplified counterparts for levels 8 and 4. In other words, the corpus contains instances where the same input leads to different outputs. Disregarding this factor may lead to suboptimal models. To avoid this problem, previous work (Alva-Manchego et al., 2017;Zhang and Lapata, 2017;Scarton et al., 2018b) has used subsets of the corpus with only certain combinations of complex-simplified article pairs, e.g. adjacent or non-adjacent pairs. This however reduces the amount of data available for training.
Learning Simplifications for Specific Target Audiences
We propose a way of making use of this information to build more informed TS models that are aware of different types of target audiences, while still making use of the full dataset for learning. Inspired by the work of Johnson et al. (2017) for MT, we add to each original instance an artificial token that represents the target grade level of that instance in order to guide a sequence-to-sequence attentional encoder-decoder neural approach (Bahdanau et al., 2015) ( §2). In a similar vein, we also annotate the coarse-grained type of operation that should be performed to simplify the original instance, under the hypothesis that certain operations are more often used to simplify into certain grade levels. Deciding on the operation is an easier problem than performing the actual operation. We rely on both gold and predicted operation types.
Learning Simplifications for Specific Target Audiences
Experiments with models built with these artificial tokens outperform state-of-the-art neural models for TS, with the best approach combining grade level and type of operation ( §3). Interestingly, such an approach also enables zero-shot TS, where a simplification for a grade level pair unseen at training time can still be generated during testing. We show that our zero-shot learning models perform virtually as well as our grade/operationinformed models ( §4). To the best of our knowledge, this is the first work to build TS models for specific target audiences and to explore zero-shot learning for this application.
Learning Simplifications for Specific Target Audiences
Our approach follows that of Johnson et al. (2017), a multilingual MT approach that adds an artificial token to encode the target language to the beginning of each source sentence in the parallel corpus. With this modified version of the corpus, a single encoder-decoder architecture is used to deal with different language pairs. Based on the tokens, the source sentences are encoded differently according to the target language they have been paired with in the corpus. Such an approach enables zeroshot MT, where a model is able to provide translations for language pairs it has not seem at training time.
Learning Simplifications for Specific Target Audiences
We apply three types of data manipulation, where artificial tokens are added to the beginning of original side of both training and test instances:
Learning Simplifications for Specific Target Audiences
• to-grade: the token corresponds to the grade level of the target instance, • operation: the token is one of the four possible coarse-grained operations that transforms the original into the simplified instance, • to-grade-operation: concatenation of the two above tokens. Different from the grade level, which can be available at test time simply by knowing the intended reader of the text, information about the operations to be performed, which we extracted from the parallel corpus, will not be available at test time. We use gold labels extracted from the parallel corpus for an oracle experiment but also use a classifier that predicts the operations for the test set based on those in the training data. We built a simple Naive Bayes classifier using the scikit-learn toolkit (Pedregosa et al., 2011) and nine features :
Learning Simplifications for Specific Target Audiences
• number of tokens / punctuation / content words / clauses, • ratio of the number of verbs / nouns / adjectives / adverbs / connectives to the number of content words. Table 1 shows examples of the tokens used when an original instance is marked to be simpli-to grade level 4 to grade level 2 to-grade <4> dusty handprints stood out against the rust of the fence near Sasabe.
Learning Simplifications for Specific Target Audiences
<2> dusty handprints stood out against the rust of the fence near Sasabe. operation <identical> dusty handprints stood out against the rust of the fence near Sasabe.
Learning Simplifications for Specific Target Audiences
<elaboration> dusty handprints stood out against the rust of the fence near Sasabe. to-grade-operation <4-identical> dusty handprints stood out against the rust of the fence near Sasabe.
Learning Simplifications for Specific Target Audiences
<2-elaboration> dusty handprints stood out against the rust of the fence near Sasabe. reference dusty handprints stood out against the rust of the fence near Sasabe.
Learning Simplifications for Specific Target Audiences
dusty handprints could be seen on the fence near Sasabe. For level 2 the reference is a rewrite and, therefore, the operation token is <elaboration>.
Learning Simplifications for Specific Target Audiences
We use OpenNMT 2 as our encoder-decoder architecture. Both encoder and decoder have two LSTM layers, hidden states of size 500 and dropout = 0.3. Global attention combined with input-feeding is used, as describe in (Luong et al., 2015). A model is trained for each dataset constructed with different artificial tokens for 13 epochs. The best model is selected according to perplexity on the development set. Figure 1 shows the architecture of the neural network, including attention and input-feeding. In this example, <token> represents the artificial token added to the pre-processed data. We evaluate our models with BLEU 3 (Papineni et al., 2002) (a proxy for grammaticality assessment), SARI (Xu et al., 2016) 4 (a proxy for simplicity assessment) and Flesch Reading Ease 5 (a proxy for readability assessment). According to Xu et al. (2016), BLEU shows high correlation with human scores for grammaticality and meaning preservation, whilst SARI shows high correlation with human scores for simplicity. Although previous work have also relied on human judgements of grammaticality, meaning preservation and simplicity, in our case such a type of evaluation is infeasible: we would need to involve judges with specific grade levels or rely on professionals who are experts in grade level-specific simplification to make such assessments.
Learning Simplifications for Specific Target Audiences
Our version of the Newsela corpus has 550, 644 instance pairs (11M original tokens and 10M target tokens), which we randomly divided into training (440, 516 instances: 80%), development (55, 064 instances: 10%) and test (55, 064 instances: 10%) sets. Instances were aligned using the method by Paetzold and Specia (2016). Xu et al. (2015) report over 56K original sentences and approximately 305K sentences including the original ones and all simplification types. Our number of instance pairs is higher because we allowed alignments from original to all simplified versions and among simplified versions. An original article 0 may be aligned to up to four simplified versions: 1, 2, 3 and 4. For each article, the alignments were extracted between 0-{1,2,3,4}, 1-{2,3,4}, 2-{3,4} and 3-4, where available. Our corpus is also larger than the ones used in (Alva-Manchego et al., 2017;Scarton et al., 2018b) and (Zhang and Lapata, 2017). While the former use only adjacent levels (e.g. 0-1, 1-2) and the latter only non-adjacent levels (e.g. 0-2, 1-4), we make use of the full dataset.
Learning Simplifications for Specific Target Audiences
As baseline we trained a model using Open-NMT and the same hyperparameters as described in §2 on the entire Newsela corpus but without artificial tokens (s2s model). The state-of-the-art model is represented by NTS, which was also trained on the entire corpus using a similar Open-NMT architecture with the same hyperparameters but additional pre-trained word embeddings as described in Nisioi et al. (2017). 6 As shown in Table 2 the NTS system performs slightly worse than the baseline system according to BLEU and SARI. Although concatenating global and local embeddings has led to improvements for the W-SW corpus in (Nisioi et al., 2017), this does not seem to be the case for the Newsela corpus. Our models outperform both the baseline and NTS systems by a large margin. Examples of outputs from all systems can be found in the Supplementary Material.  The best model is the one built with the <to-grade+operation> token with gold operations annotations (last row). The second best system uses the gold <operation> token only. Therefore, knowing the operation type to be performed for a given instance provides valuable information. Even though the models with predicted operations ('pred' in Table 2) still outperform the baseline, they lag behind their counterparts built using gold operations. The main reason for that is the very simplistic classifier we used (average accuracy = 0.51, calculated using 10-fold crossvalidation). In summary, s2s+to-grade is the best performing model in a real world scenario, given the low performance of 'pred' systems. A more informed classifier should lead to better results, but this left for future work; our goal was to show the potential of this information.
Learning Simplifications for Specific Target Audiences
The improvements in SARI are substantial: 7 points over the baseline even with the predicted operations. However, SARI aims to measure simplicity in general (not for specific grade levels). Since human evaluation of the targeted simplification performed by our models is not feasible, we can only approximate the usefulness of our models by using readability metrics such 6 Equivalent to their best performing "NTS-w2v" version. as the Flesch-Kincaid Grade Level. This metric maps a text into a US grade level, which is the same grading provided in the Newsela corpus and, therefore, relevant for our study. Table 3 shows the Flesch-Kincaid results for the test set divided into the appropriate grade levels considering the outputs of s2s, s2s+to-grade and s2s+to-grade+operation (gold) models. Simplifications generated by s2s+to-grade and s2s+to-grade+operation are scored consistently closer to the appropriate grade, which does not happen with s2s.  Table 3: Flesch-Kincaid scores for instances of each grade level simplified using s2s, s2s+to-grade and s2s+to-grade+operation (gold) models.
Learning Simplifications for Specific Target Audiences
The last row of Table 3 shows the Mean Absolute Error (MAE) considering the Flesch-Kincaid Grade Level scores for the system outputs as the hypothesis and the expected grade level as the gold scores. Our s2s+to-grade and s2s+to-grade+operation (gold) models show lower error scores than the baseline system, which supports our hypothesis that such models produce more adequate outputs for targeted grade levels.
Learning Simplifications for Specific Target Audiences
The main advantage of s2s+to-grade is that a user can inform their grade level and retrieve a personalised simplification. Table 4 shows an example with different simplifications for an out-of-domain instance from the SimPA corpus (Scarton et al., 2018a). The same instance was given as input to the s2s+to-grade model with different artificial tokens according to the grade level that we want to achieve. The s2s system (second row) repeats the original instance (first row). Conversely, our s2s+to-grade model is capable of distinguishing among different levels and produces personalised simplifications for each grade level.
Learning Simplifications for Specific Target Audiences
original We want to reassure you that we take fire safety very seriously and we are doing everything we can to make sure our residents are safe. s2s
Learning Simplifications for Specific Target Audiences
We want to reassure you that we take fire safety very seriously and we are doing everything we can to make sure our residents are safe.
Learning Simplifications for Specific Target Audiences
We want to reassure you that we take fire safety very seriously and we are doing everything we can to make sure our residents are safe.
Learning Simplifications for Specific Target Audiences
We want to reassure you that we take fire safety very seriously and we are doing everything we can to make sure our residents are safe.
Learning Simplifications for Specific Target Audiences
We want to reassure you that we take fire safety very seriously and we are doing everything we can to make sure our residents are safe.
Learning Simplifications for Specific Target Audiences
We want to reassure you that we take fire safety very seriously and we are doing everything we can to make sure our residents are safe.
Learning Simplifications for Specific Target Audiences
We want to reassure you that we take fire safety very seriously. We are doing everything we can to make sure our residents are safe.
Learning Simplifications for Specific Target Audiences
We want to reassure you that we take fire safety very seriously. We are doing everything we can to make sure our residents are safe.
Learning Simplifications for Specific Target Audiences
We want to make sure we take fire safety very seriously. We are doing everything we can to make sure our people are safe.
Learning Simplifications for Specific Target Audiences
We want to make sure people take fire safety very seriously. We are doing everything we can to make sure our people are safe.
Learning Simplifications for Specific Target Audiences
We want to make sure people take fire safety very seriously. We are doing everything we can to make sure people are safe. 
Learning Simplifications for Specific Target Audiences
To show that zero-shot TS is possible, we build models on training data without instances of a certain grade level pair and test them on instances of that grade level pair. Consider the grade level pair < g o , g t >, where g o is the grade level of an original instance o, g t of a target instance t, and t is aligned to o. We test if our "s2s+to-grade" model can generalise for instances of <ĝ o ,ĝ t > that have not been seen at training time.
Learning Simplifications for Specific Target Audiences
Due to space restrictions, we only show results for three representative grade level pairs. These pairs have a large enough number of training and test instances and cover levels that are closer or further apart from each other. In addition, after removing them the training corpus still has enough instances of theĝ t as target grade level. Instances of the target but not the original level (or of the target language in MT) must exist for zero-shot to be possible. The distributions of the selected grade level pairs is shown in Table 5  In Table 6, the s2s and s2s+to-grade models are the same as in Section 3, i.e. trained with the entire dataset without artificial tokens (s2s) or with artificial tokens (s2s+to-grade). The zero-shot models (s2s+to-grade+zs) are trained with <to-grade> data, but after removing instances of the grade level pair <ĝ o ,ĝ t > under investigation, i.e. on a smaller dataset. For < 12, 7 > and < 12, 4 >, the zero-shot models outperform the baseline according to all metrics. In terms of SARI, for < 12, 7 > the zero-shot model is only marginally worse than the s2s+to-grade model. Conversely, s2s+to-grade+zs outperforms s2s+to-grade for < 12, 4 >, which is an impressive result. Finally, for < 6, 5 > all three models perform similarly. This may be explained by the proximity ofĝ o andĝ t , which means that instances must be considerably close to each other and therefore simplifications will be minor and have little impact in the scores.  
Learning Simplifications for Specific Target Audiences
We have presented an approach for TS that benefits from corpora built for various target audiences and allows building better models than generalpurpose ones. We have also shown that zero-shot learning is possible for TS, where instances of the original-target audience do not exist. As future work we intend to investigate (i) better classifiers to predict operation types and (ii) multi-task learning as an alternative way of building a single TS model for various specific target audiences. We also plan to run experiments with the W-SW corpus and using an improved classifier to train models with information on operations.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
Automatic text simplification (ATS) is a task that consists in automatically adapting a text to make it more accessible for readers. It is the focus of more and more attention from researchers, with a growing number of publications every year and some surveys published recently (Saggion, 2017;Al-Thanyyan and Azmi, 2021;Štajner, 2021). As it is the case in many domains, ATS tasks are currently mostly explored using deep learning (Nisioi et al., 2017;Alva-Manchego et al., 2020b;Cooper and Shardlow, 2020), although rule-based systems are still being explored as well (Saggion, 2017;Evans and Orasan, 2019;Wilkens et al., 2020).
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
One serious hurdle that the field is currently facing is how to evaluate those systems (Grabar and Saggion, 2022). There are two common approaches: human judgment and automatic evaluation metrics. Human judgment is collected by asking people to rate the output of a system on a Likert scale using three criteria: grammaticality, meaning preservation and simplicity. This method is certainly encompassing, but requires lots of time and effort. In contrast, automatic evaluation metrics represent a reproducible and quick way to measure the performance of ATS systems. The most common metrics currently used are: BLEU (Papineni et al., 2002), a metric created to evaluate the performance of machine translation systems; SARI (Xu et al., 2016a), a metric specifically designed for ATS; and the Flesch-Kincaid readability formula (Kincaid et al., 1975). Although these metrics have serious shortcomings (Sulem et al., 2018a;Alva-Manchego et al., 2021;Tanprasert and Kauchak, 2021), their use is widespread in the field due to their ease of use. BLEU and SARI require reference data (human made simplifications) and are known to be more reliable when more references are available. In other words, their behavior is a function of the dataset used as the reference. Since different audiences have different needs when it comes to simplification (Rennes et al., 2022), making sure that a dataset will reflect those specific needs is important. Two other automatic metrics have been used in text simplification research but are not broadly used: BertScore (Zhang et al., 2020) -a metric designed for language generation evaluation -and SAMSA (Sulem et al., 2018b) -specifically designed for ATS but not easy to use as it requires semantic annotation.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
In English, mostly three corpora are used for evaluation: TurkCorpus (Xu et al., 2016b), ASSET (Alva-Manchego et al., 2020a) and Newsela (Xu et al., 2015). The first two share the same source sentences, with different crowdsourced simplifications. For other languages, with less attention from the community, systems are evaluated against ad-hoc corpora (Kodaira et al., 2016;Cardon and Grabar, 2020;Anees and Abdul Rauf, 2021;Spring et al., 2021). Little is known about the way automatic metrics are related to the different types of simplification operations. Recent work (Vásquez-Rodríguez et al., 2021) investigated the relation of those metrics to basic computational operations such as add, delete or insert, but no work focuses on the linguistic simplifications listed in existing typologies (Brunato et al., 2022;Gala et al., 2020;Amancio and Specia, 2014).
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
In this paper, we aim to investigate the following hypothesis: evaluation metrics react differently depending on the type of linguistic operations applied to produce the reference(s) and the output of a system. To verify this hypothesis, we have annotated the ASSET corpus with a broad set of linguistic operations. The annotated resource is the first contribution of the paper and we hope it can spur research on new automated evaluation metrics. A second contribution is the annotation process itself (see Section 3), as we provide an annotation guide that has been validated with 9 annotators and could be reused to annotate other corpora in a similar way. In Section 4, we study SARI's behavior according to the operations we annotated and we calculate the SARI and BLEU scores of published systems on various subsets of ASSET (Section 4.2). This new angle of observations on automatic ATS evaluation, in relation with our hypothesis, is the third contribution of this paper.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
Historically, before statistical and then neural methods became the main focus of ATS research, typologies were a requirement for building ATS systems, as they were the conceptual basis of rule-based methods. As documented by Siddharthan (2014), authors of ATS systems were mostly concerned with syntax (Chandrasekar et al., 1996;Dras, 1999;Brouwers et al., 2014) and produced typologies of the syntactic operations they aimed at performing on sentence structures. In more recent works, we could identify two main axes that were used to build typologies of simplification operations: one based on linguistic descriptions and the other one based on string edits.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
The first type of typologies aims at identifying the linguistic operations at work during the simplification process. They have been studied for a variety of languages: Spanish (Bott and Saggion, 2014), Italian (Brunato et al., 2014(Brunato et al., , 2022, French (Koptient et al., 2019;Gala et al., 2020), Brazilian Portuguese (Caseli et al., 2009, Basque (Gonzalez-Dios et al., 2018) and English (Amancio and Specia, 2014). While they share some operations, for instance the transition from the verbal passive voice to the active voice, those typologies also have specific categories such as proximization (Bott and Saggion, 2014) -changing a sentence in order to address the reader directly -or specification (Koptient et al., 2019) -keeping a difficult term but adding an explanation next to it. Interestingly, these two examples depend on the genre of the text: it is unlikely to use proximization when simplifying encyclopedia articles for instance, and it is expected that specification would occur in text genres that involve technical jargon. Those typologies have been used to give more information about simplification corpora, and to observe what humans do when they simplify texts. It should be noted that none of the linguistically-based typologies have been integrated into an evaluation protocol for ATS systems, which is something that we hope to enable with this work.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
This axis pertains to operations where sentences are seen as strings of tokens, that are edited during simplification. To the best of our knowledge, it has been explored almost exclusively for English (Coster and Kauchak, 2011;Alva-Manchego et al., 2017, 2020aVásquez-Rodríguez et al., 2021), with one recent exception for Italian (Brunato et al., 2022). As with linguistic typologies, the operations also differ from one work to the other, but the approach always consists in observing what happens to the tokens during simplification. There have been multiple uses for this king of typology. Like the linguistically-based one, it has been used to perform corpus analysis as a goal in itself (Alva-Manchego et al., 2020a). It has also been used to study the relation between string distances and the scores given by automatic evaluation metrics (Vásquez-Rodríguez et al., 2021). Some ATS systems incorporate such operations in their architecture (Alva-Manchego et al., 2017;Dong et al., 2019;Agrawal et al., 2021). The evaluation metric SARI integrates such operations as sub-components in its computation: keep, add and delete (See section 4.1 for more details about this).
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
This section presents the dataset (Section 3.1) and the typology of the operations we use for annotation (Section 3.2). We then make a statement on what we chose not to annotate but could have been expected given the data and the current practices in ATS evaluation (Section 3.3). We describe the annotation process (Section 3.5) and we analyze the annotated set that served for the inter-annotator agreement (Section 3.5), then we describe the resulting resource (Section 3.6. We finally compare our result with other similar works (Section 3.7.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
Two freely available corpora are currently widely used for evaluation in works on ATS for English: TurkCorpus (Xu et al., 2016b) and ASSET (Alva-Manchego et al., 2020a). We retained ASSET, which has been independently described as an improvement over TurkCorpus (Vásquez-Rodríguez et al., 2021). The following section introduces the operations with which we annotate the 3,590 references in ASSET's test set.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
The works presented in Section 2.1 propose typologies built upon corpus observations. We build our own, relying on those typologies. In consequence, no new operations are introduced. The main goal is to have a typology that can be used as the core on which to build to analyze any corpus in any of the languages on which our source material is made. We did not take into account the genre-specific operations, such as the ones mentioned in Section 2.1 (proximization and specification). For convenience, we do not use a fine-grained distinction between the grammatical functions of the constituents that we annotate, thus we merged all the different partof-speech modifications into a single operation. Below we present the resulting set of operations. Each operation is shown along its short name that we use in the rest of the article. Some of them are self-explanatory and some are clarified with a short comment. For more details and examples, see Section 3 of the annotation guide in Appendix A.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
We introduce the operations in two distinct sets: the first one contains operations that can be mapped to computational operations. The mapping of those operations to computational operations is the following: insert (also referred to as add in the literature), delete and move are already words that are used for computational operations. All the other categories are replacements/substitutions. It should be noted that the computational operation called replace is sometimes considered as a combination of add and delete, as it is the case in SARI for instance.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• Move (move) • Insert/Delete proposition (inprop, delprop)
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• Insert/Delete modifier (inmod, delmod). The definition of modifier we use is quite loose. This covers both word-level modifiers (e.g., a qualifying adjective modifying a noun) and sentence-level modifiers (e.g., adverbial phrases).
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• Insert/Delete for consistency (incst, delcst). Any insertion or deletion required for the sentence to remain grammatical after another operation is performed.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• Insert/Delete Other (inoth, deloth). Any insertion or deletion that does not fit in the other insert categories.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• Replace with synonym (synonym)
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• Replace with hyperonym (hyperonym)
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• Replace with hyponym (hyponym)
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• Replace singular with plural (s2p)
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• Replace plural with singular (p2s)
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• Replace segment with a pronoun (pron)
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• Replace pronoun with its antecedent (fromPron)
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• Modify verbal features (verbf). Any change of modality or tense on a verb.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
The second set contains operations that can be performed with various computational operation combinations, or that are too complex to be consistently mapped to computational operations.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• Active to passive (a2p)
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• Passive to active (p2a)
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• Part-of-speech change (POSchange)
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• To personal form (fromImp)
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• Affirmation to negation (a2n)
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• Negation to affirmation (n2a)
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
We also added a label named Erroneous simplification (err). While we do not assess simplicity, this label lets the annotators signal manifest errors either in adequacy or grammaticality that make the simplification irrelevant as a reference in the evaluation of an ATS system.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
After describing the data we use and the typology that we propose, we would like to mention the aspects that we chose not to annotate. Usually, manual evaluation of simplifications is performed on three criteria: adequacy or meaning preservation, fluency or grammaticality, and simplicity. As the simplifications in the corpus were made by humans, we do not use the usual 5-point Likert scale and we simply chose to mark sentences that had obvious issues in one of those aspects as erroneous. We chose not to assess simplicity at all for several reasons.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
First, in the literature the methods for doing so are not consistent from one work to the other; there is a need for standardization (Stodden, 2021). We consider that this is out of the scope of this paper. Second, we believe that a judgment on simplicity should be made by members of the target audience of the corpus or the system that is evaluated. ASSET was not made for an identified target audience, and, as an NLP research team, we do not represent a typical demographic target. Plus, Alva-Manchego et al. (2020a) collected human judgments of simplicity on 100 sentences from ASSET via crowdsourcing (with no specific demographic target), and obtained an inter-annotator agreement (Cohen, 1968) of 0.628. That observation corroborates the claim that assessing simplicity as a textual property, without a target audience in mind, is not optimal (Gooding, 2022).
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
The annotation was performed using YAWAT (Germann, 2008), which has been used for the same purpose previously (Koptient et al., 2019). YAWAT lets the user create groups of tokens to annotate within sentence pairs, either belonging to both sentences or to one of the two (typically, insertions and deletions occur only on one side, while other operations such as synonymy involve tokens from both sides). 1 The creation of the typology preceded the start of the annotation process and was the basis for writing the annotation guide. Four persons (all NLP researchers: two PhD students and two postdocs) annotated the same 50 sentence pairs from ASSET. Three work directly in text simplification, one in adjacent domains such as lexical complexity 1 A more recent and more user-friendly tool for ATS corpora annotation exists, TS-ANNO (Stodden and Kallmeyer, 2022) but was not available yet during our annotation. or readability. That step was used (1) for assessing the clarity of the annotation guide, (2) to train the annotators on the annotation tool and (3) to discuss how to improve the annotation guide. It was the occasion to discuss difficult cases and how to address them 2 in order to reach consensus, and to tune priority rules. The typology itself was not modified as a result of those discussions.After this, we estimated that the annotation requires 10 hours per 100 sentence pairs as an upper bound. We reproduced that step twice, with 25 sentence pairs each time.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
Once the final version of the guide was created, we hired five Master's students in linguistics (all enrolled in an NLP track) for the annotation 3 . They went through the same last two steps described above. The students were paid 4 to annotate. They could perform the task at their convenience. The final step before the whole annotation began was to have everyone -the four researchers and the five students -annotate the same 50 new sentence pairs from ASSET. We calculated the inter-annotator agreement with the Davies & Fleiss agreement (Davies and Fleiss, 1982) using each token's label. The score was computed separately for the source side and the target side. This resulted in an agreement of 0.61 for the source side and 0.68 for the target side. We also calculated the agreement with all the different insertions merged into one category, and all the deletions merged into one category. This increased the agreement scores to 0.74 for the source side and 0.72 for the target side. This indicates that there is a tradeoff between the granularity of the labels and the agreement that can be expected. The first author manually reviewed the nine resulting sets of 50 sentence pairs in order to produce a single dataset. We did not proceed automatically in order to avoid any inconsistencies coming from disagreements. This aggregated dataset is referred to as the gold dataset in this paper, and the fully annotated dataset is referred to as ASSET ann .
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
In this section, we compare the annotations of each annotator versus the reference, on the gold dataset. First, the average κ score of each annotator with the gold is 0.73 for the original sentences and 0.74 for the simplified sentences, which corresponds to a substantial agreement. Moreover, the standard deviation of the 9 annotators is rather low (σ = .04 for the source and σ = .05 for target), with κ values ranging from 0.65 to 0.79. This confirms that all annotators were able to annotate with a rather similar level of reliability.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
In order to give a view on the reliability of the annotation per operation, we computed recall and precision by comparing the annotations of each annotator to the gold, per label. We averaged those values over the 9 annotators to get a global view of the categories that are difficult to annotate (see Table 8 in Appendix C for the complete results).
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
For most categories, the recall and precision values seem satisfactory, being superior to 0.6, which is in line with the global robust κ. On source sentences, annotators were prone to forget about the fromPron category, whereas they had trouble to correctly identify hyponyms and POS changes. A close investigation of the confusion matrices of each annotator reveals that most errors regarding hyponyms and hyperonyms are due to their annotation as synonyms. On simplifications, annotators tended to miss the two categories fromPron (confused with incst about 50% of the time) and toImp. In addition, they had trouble to correctly use the following categories: inoth (often confused with another insert operation such as incst and inmod) and inprop (which is also mixed with other insert operations).
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
This section describes the resource, called ASSET ann , resulting from the annotation process. The ASSET test set contains 3,590 pairs of original and simplified sentences (359 sentences with 10 simplifications each). During the annotation process, we observed 19 pairs of identical sentences, and 227 erroneous simplifications across 157 different source sentences. 5 ASSET ann contains 3,323 annotated pairs of original and simplified sentences. In total, 12,827 operations were identified. Synonym is the most common operation observed (14% of the total number of operations). In fact, seven operations (synonym, delcst, deloth, incst, delmod, move and delprop) account for 70% of the operations in both the gold and ASSET ann . 5 4 source sentences with 4 errors, 10 source sentences with 3 errors, 32 sentences with 2 errors and 105 with 1 error.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
The number of operations in both datasets can be seen in Table 1. Notably, 17 operations were observed less than 10 times in the gold corpus.In the whole dataset, we identified that 31 annotations with delete labels (out of 5 073) were in the simplified side and 52 annotations with insert labels (out of 2 547) were in the original side. As those are errors, we automatically changed those labels into their adequate counterpart (e.g. changing all delcst labels found on the simplified side into incst operations). Table 1 and the results reported in Section 4 take this adjustment into account. We did not apply this to the data used for the analysis in Section 3.5. In order to see whether the number of operations in a sentence can be affected by the sentence length, we analyzed the relation between those two aspects (see in Appendix B for details). We found that while the longest sentences show the highest operation count, there is no significant correlation between the number of operations and the length of a sentence.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
Another important perspective is the number of operations per sentence. Given the nature of text simplification, different levels of linguistic operations are expected (e.g., lexical and syntactical operations). In this sense, we observed that 13% of the simplified sentences are the result of a single operation. Moreover, 50% of the sentences in the corpus result from up to 3 operations, and 88% of the sentences have up to 6 operations (see Table 7 in Appendix B).
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
Finally, we identified the number of tokens added and deleted by each operation (see Table 6). This enables to verify the mapping between linguistic and computational operations. As expected, all deletions tend to remove all the annotated words. However, in some cases, some words remain, or other words are inserted, in order to keep the sentence correct. Similarly, fromImp and pron operations tend to remove tokens in the sentence. In the opposite direction, content insertion is most remarkable in a2p, fromPron, neg2pos, and pos2neg operations, and the operations specific for insertions (incst, inmod, inoth and inprop). The other operations do not modify the number of tokens. Comparing the token count in both sides of the corpus in relation to each operation, we observed that all operations could be arranged as add, del and replace following the guide. 6 This analysis allows us to indirectly measure the quality of the annotation, and it also allows us to observe that 25% of the operations in ASSET are insertions, 40% are deletions and 35% are replacements.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
Some works proposing typologies mentioned in Section 2.1 report proportions for the operations they annotated in their respective corpora. In this section, we compare those observations. It is important to note that all the compared works vary regarding the corpora they use, namely along characteristics such as context of creation, language, size, text genre, domain and target audience. The typologies that were used are all different as well.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
As overcoming all those gaps for a comprehensive analysis is out of the scope of this paper, we propose a simple overview of the results. Table 2 shows the outcome of our comparison. We include information about the context of the corpus creation in the Method column. Indeed, some source corpora were created as part of a work on ATS (S), and others were not (A). This might have an influence on the simplification process. For example, ASSET's crowdworkers were given examples of original sentences and simplified versions before performing the task. As one of the goals of the creators of ASSET was to produce a parallel corpus with "multiple rewriting operations" -in opposition to TurkCorpus which contained lexical operations -the approach may explain why the proportion of lexical operations we found is the lowest. The highest proportion of lexical operations was found in the CLEAR corpus. As CLEAR was built by aligning sentences from comparable corpora in the medical domain, this may be explained by the amount of specialized terminology in the complex side. The CBST corpus displays two methods of corpus creation, called "Intuitive" and "Stuctural". In both cases, the approach was to simplify scientific texts so that they could be understood by children. The first one consisted in asking a teacher to rewrite texts following only their knowledge of the target audience, the other one consisted in asking a court translator to simplify using easy-to-read guidelines. While the proportion of syntactic operations is similar in both cases, the other types vary.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
Those surface observations indicate that there might not exist a universal way of simplifying a text, even within a given language. The factors that we could identify are numerous: language, target audience, domain, genre, profile of the person(s) performing the simplification, context of the simplification task (for human readers or for research), and type of instructions given. We believe that those criteria should be taken into account when working on ATS systems during the design of the system, the choices made in the selection and preparation of the data, the evaluation protocol and the comparison with other works.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
We propose two experiments that show how ASSET ann can help studying evaluation protocols. First, we analyze the behavior of SARI, with respect to ASSET ann . Second, we re-evaluate existing ATS systems from the literature, to have a view on how they handle linguistic operations.  Table 2: Comparison of reported simplification corpora contents. In the Method column, A = aligned from a pre-existing corpus (either comparable or manually simplified), S = manually simplified within a work related to ATS. In the Size column, sp = sentence pairs, dp = document pairs. The reported size corresponds to the sample used for analysis, which may differ from the actual size of the corpus with the same name. A cell containing "-" means either that the value is 0 or that the information was not reported for this operation type.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
We use ASSET ann to analyze SARI's and its subcomponents' behavior in relation to simplification operations. SARI is composed of three subcomponents that are averaged to obtain SARI's final score. These sub-components are keep, add and del, respectively taking into account n-grams that are kept, added or deleted from the original sentence to the simplified one, taking reference(s) into account. More precisely, for each sub-component (sc) keep, add and del, the F1-score is computed for each n-gram size n 7 :
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
We created a tabular dataset in which the presence of operations is represented by the number of occurrences in the sentence pair annotation. For instance, "move = 2" and "synonym = 0" means that two move operations were applied to the sentence to obtain the simplification, while no synonym operation was found. By observing the correlation between the presence of specific operations and the global SARI scores, we find little correlation between the two. This means that no specific operation seems to correspond to the global SARI score. What is surprising is that operations mapped to SARI's sub-components (insertions and deletions) are not correlated with SARI scores (see Table 9 in Appendix D), despite being correlated with SARI's sub-components (see Table 10 in Appendix D).
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
In order to go further, we analyze how well combining operations can predict SARI's score. To 7 This description of SARI is the one implemented in EASSE (Alva-Manchego et al., 2019), which is the implementation we used throughout this paper. evaluate our models, we use an average R 2 using a 10-fold cross-validation, with 9 folds for training and 1 fold for testing. The 157 source sentences for which we found at least one erroneous simplification were removed from the experiment. This is to make sure that all original sentences contain 10 non-erroneous simplifications for the 10-fold cross validation procedure. A Lasso regression model (Tibshirani, 1996) with optimized hyper-parameters barely obtains a R 2 of 0 when predicting the final SARI score. This indicates that the model cannot predict better than by simply using the mean. We found the same result with regression trees (Breiman et al., 1984), random forests (Breiman, 2001) and multi-layer perceptrons (Hinton, 1989) 8 . However, predicting SARI's sub-components is at least possible using Lasso with an average R 2 of 0.24, 0.03 and 0.23 for keep, add and del respectively. Table 3 presents the coefficients of a Lasso model trained on the whole corpus to predict SARI's sub-components. These coefficients were obtained with an R 2 of 0.25, 0.05 and 0.24 for keep, add and del respectively. The main finding at this level is that a large amount of operations have 0 as coefficient, meaning those have no effect on SARI's sub-components.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
We can state that while SARI has a degree of relation to linguistic operations, averaging the score of all its sub-components hides this piece of information. This highlights two issues with SARI. First, SARI has a very low variance and is not very sensitive to the differences between system outputs. This makes predictions using only SARI's mean really efficient, as averaging three very different sub-components always leads to roughly the same SARI score. Second, as SARI relies on references  to compute its score, the average on several references (in our case 9 references) reinforces the low variance of SARI. This last issue is also present for the three sub-components of SARI, which explain the somewhat low R 2 scores. Indeed, in all cases, predicting with the mean already makes it possible to obtain good results. It is therefore difficult to find the particular operations that explain more than what the mean already explains. This is particularly true for the add sub-component. This interpretation also accounts for the reason why the Lasso coefficients in Table 3 make sense, despite their associated low R 2 : it is very difficult to explain what the mean does not already explain.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
Using ASSET ann , we highlighted two elements of SARI. First, we presented in Table 3 the operations that best match the sub-components of SARI. This favors the use of SARI's sub-components for evaluation, instead of the global score. Second, the difficulty to make better predictions than the mean highlights the issue of SARI's low variance.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
This section shows how the annotation we propose can be useful for the ATS research community. First, as we identified 2 % of erroneous simplifications -which might affect the current perception of existing systems -, we rescored the systems proposed by Zhang and Lapata (2017)  We also analyzed the effect of linguistically motivated groups of operations. For that, we created a subset of ASSET by keeping only references that were annotated with at least one lexical operation (i.e., synonym, hyponym, and hyperonym -2,138 references) and another subset with references having at least one syntactic operation (i.e., inprop, inmod, inoth, delprop, delmod, deloth, a2p, p2a, split, toImp, fromImp, a2n, n2a, -2,799 references). These results are presented in columns synt and lex of Table 4. In general, SARI scores decreased for all models. MUSS wiki ♠ (Martin et al., 2020) remains the highest scoring system for lexical and syntactic simplification. In general, we observed that the decrease of SARI is on average 1.3 with the syntactic subset, and an impressive decrease average of 3.5 points with the lexical subset.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
Finally, we explore the behavior of computational operations in relation to our mapping (see Section 3.2) -also observed in the number of added, deleted, and replaced tokens (see Section 3.6) -in order to observe the systems' performance. We create 3 subsets: one with references containing at least one add operation (2,040 references), another one with del operations (2,866 references) and one with replace operations (2,719 references). Comparing the SARI scores, MUSS systems tend to present similar scores for the three subsets, but the difference between the scores in the entire dataset (column ORIG) and each subset is remarkably different: MUSS newsela ♠ is the most stable system with a decrease of 2.7, 0.5 and 1.4 respectively for add, del and replace subsets. On the other hand, MUSS wiki is the system with the greatest loss of performance 4.3, 1.5 and 2.0 respectively for add, del and replace subsets. In general, the systems kept a similar SARI score with del (average decrease of 1.3) and replace (average decrease of 1.8) subsets. Using the add subset, SARI decreased on average by 4 points on all systems. It is noticeable that systems with better SARI scores have a large proportion of add.  
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
We performed the annotation of ASSET with the goal of shedding light on automatic ATS evaluation using linguistic information. This enabled us to make several insightful contributions.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
Regarding the analysis of current automatic evaluation, we have shown that SARI's subcomponents can inform on the linguistic operations present in references that appear in a system output, whereas this is lost when averaging the three subcomponents into one single score. This insight is an argument in favor of reporting the sub-components' scores while performing evaluation, as some works started to do (Zhao et al., 2020;Tanprasert and Kauchak, 2021). The analyses we performed also make the case for further exploration in bridging linguistic operations and computational operations. We believe that ASSET ann and our experiments can serve to pave the way for future ATS evaluation practices that would go further in that direction.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
We selected subsets of references to use for the evaluation according to predetermined sets of operations. Creating such subsets allows to focus on relevant operations for a given target audience's specific needs in the evaluation framework. ASSET ann is a first step in that direction.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
As discussed in Section 3.3, we did not evaluate simplicity. Though it can be time consuming, we recommend to assess simplicity based on the precise definition of the task. During the resource 2% of the sentences identified as erroneous simplification. creation, annotators told us they thought that some simplifications were, in fact, obviously not simpler. Working with members of the target audience of a given task in order to identify references that should not be taken into account would allow for a more accurate evaluation protocol.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
Finally, as a future work we would like to leverage the annotated dataset to automatically reproduce the annotation.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
In this work, we produced an annotated version of the ASSET test set. The resource was annotated by 9 annotators, following a typology that we proposed based on existing similar typologies. Based on the annotation, we could clean the test set by removing 227 sentence pairs with manifest errors and produce a curated version of ASSET. We reevaluated systems for which the outputs were available and provided the community with updated performance results using SARI and BLEU. The different resources described in the paper and the code used for our analyses are available online. 10 In addition, we performed an extensive analysis of SARI's sub-components and found links between those and linguistic operations. We see these results as a promising direction to improve automatic ATS evaluation by exploring the relationship between computational and linguistic operations.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
We would like to express our gratitude to Nils Bouckaert, Elena Cao, Angela Kasparian, Melanie Johanns and Luca Matarelli, who helped us annotate the dataset. We also thank Damien de Meyere and Hubert Naets for their support with YAWAT.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
We also thank the anonymous reviewers for their suggestions and comments that helped improve the quality of the paper.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
Rémi Cardon is supported by the FSR Incoming Postdoc Fellowship program of the FSR -Université Catholique de Louvain. Adrien Bibal is supported by the Walloon region with a Win2Wal funding. Rodrigo Wilkens is supported by a research convention with France Education International (FEI). David Alfter is supported by the Fonds de la Recherche Scientifique de Belgique (F.R.S-FNRS) under the grant MIS/PGY F.4518.21.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
The limitations of our work are the following:
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• We do not comment upon simplicity in the corpus. The reason for this is explained in Section 3.3;
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• We should mention that the typology proposed, as it aims to be as generic as possible, covers a large amount of simplification transformations, but might need some adaptations or additions to be applied to specific contexts or languages;
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• We identified that some operations were less consensual to annotate than others and that granularity makes the task more difficult (Section 3.5). That said, this does not invalidate our analyses and we intend our typology to serve as a basis for producing task-specific sets of operations, based on the translation of users' needs into concrete simplification operations;
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• The annotation was performed only on one evaluation dataset. We discuss this in Sections 1 and 3.1. We found linguistic operations that do not influence the score of SARI's sub-components (Section 4.1), but as we only annotated ASSET's test set, more observations need to be performed to confirm or refute this conclusion. 
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
Yawat is the annotation tool used. It is a web interface. As Yawat presents security risks, access is done in two steps. First, navigate to REDACTED URL. You will be asked for a login and a password:
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• login: REDACTED
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• password: REDACTED Normally, you can save this information in the browser so that you don't have to copy the password each time.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
You then reach the Yawat connection interface where you will use the login and password communicated to you.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
Once connected, you will see a list of files to annotate. Click on a file name to go to the annotation interface proper.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
In the annotation interface, sentences are grouped: the original sentence on the left and the simplified sentence on the right.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
It it possible to switch to a different view where the sentences are shown one above the other by clicking on the logo with the two rectangles in the top right corner of the page.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
On first opening of an annotation file, check a "done" case, then uncheck it, then reload the page. Without this step, the interface will not be responsive.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
Annotating a segment of text, either on one side only (e.g., for insertions or deletions), or by aligning segments of the two sentences (e.g., for lexical substitutions) is done in three steps:
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• Click on each token of the group to annotate • Hold down Shift and click on one of the highlighted words, then release Shift. This step validates the group.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• Hold down Shift and click on one of the words in the group and the annotation menu will appear. The list changes based on whether the words of the groups appear in the two sentences or just in one of the two sentences.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
After annotating a sentence pair, you can check the "done" case. This updates your progress on the welcome page, saves the annotation of the sentence and advances the view so that the next unchecked (i.e., sentence to annotate) appears next on screen.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
To make sure that your progress is saved, click on the "save" button in the top right corner of the page.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
This section describes each element of the typology.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
This label is used to indicate words that change positions but are not otherwise modified. This label is to be annotated on the level of the constituent that is moved. Transformation operations take precedence over this label.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
Example:
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
This label is to be used with parsimony: the label is used to indicate cases where the simplification is contradicting the original sentence or where the simplification is clearly unrelated to the original sentence.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
In this case, no other label than this one is to be used; all other transformations are ignored. The label is applied to the first word of each sentence. Examples:
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
The segment to fit the baby doesn't correspond to any information in the original sentence.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
The two sentences are contradictory.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
Deleting the affixes -dependent and S-distorts the meaning of the original sentence.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
While it is possible to reconstruct the meaning of the sentence, there's a missing word between the and of in the simplified sentence.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
cst stands for consistency. This label is used for transformations imposed by the context, i.e. transformations that become necessary for the sentence to remain grammatical due to other transformations. For example, if eats becomes eaten by in a pair of sentences, by is annotated as insert-cst. Example (insertion):
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
In the example, the annotation indicates that She and was were added due to another transformation (the sentence splitting). These labels will be used abundantly, including:
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• All insertions and deletions of puncutation (except sentence splitting, see section 3.5.5)
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• All insertions and deletions of possessive markers ('s)
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• In cases where a preposition or determiner could be annotated with the synonymy label (on March 9, 2000 → in 2000; a → the)
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
mod stands for modifier. All added or deleted modifiers are to be annotated with this label. This covers both word-level modifiers (for example a qualifying adjective modifying a noun) and sentence-level modifiers (for example adverbial phrases). Example (deletion):
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
prop stands for proposition. All added or deleted propositions are to be annotated with this label. Example (deletion):
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
This label is used for all insertions and deletions not covered by the aforementioned cases. Examples (deletion):
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
_______
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• The label synonym is used in the broadest sense; verbal paraphrases are also covered by this definition, as shown in the first example.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
Examples :
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
The labels hyperonym and hyponym are rather self-explanatory. The simplified term is characterized with regards to the original term. For example, if the original sentence contains cat and the simplified sentence animal, the label will be hyperonym. These labels do not only cover what could be found in a lexical network but have to be evaluated in the context of the sentence, as shown in the following example.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
Example (hyperonymy):
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• singular to plural and plural to singular are rather self-explanatory. The simplified term is characterized with regards to the original term. If, for example, the original sentence contains cats and the simplified sentence cat, the label will be plural to singular. This label is to be used solely if the transformation is a change in number. Lexical substitution labels take precedence over this label.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
No examples encountered during the testing phase.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
There are two types of part-of-speech changes:
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• Transformations related to coreference: pronominalization and pronoun resolution. The simplified term is characterized with regards to the original term. The notion of pronoun is used in the sense of anaphoric expression, as illustrated in the following example:
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• Part-of-speech changes: POS change. This label concerns words that are in a relationship of derivation. For example, this label is used for transformations such as advertisement → advertise.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
No examples encountered during the testing phase.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
This label indicates changes in tense or modality of the verb. For example, if the original sentence contains does and the simplified sentence will do, this change will be annotated as verbal features. Lexical substitution and grammatical voice take precedence over this label. Example:
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
These labels indicate a change in voice. Only the verb is annotated with this label; changes in syntactic function of the agent or patient are not annotated. If the subject of the verb changes, this label takes priority. If the subject of the verb does not change, lexical transformations take precedence over this label. Examples:
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
The complement "the town charter" becomes the subject in the simplification, thus this label is applied.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
Since the subject is the same in both sentences, the label synonym is used instead.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
This label indicates transformations such as It is our house that the cat is in → The cat is in our house.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
In this example, It, is, that in the original sentence are annotated with the label To personal form. In the opposite case, the same words (in the simplified sentence) would be annotated with To impersonal form.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
Example (to impersonal form):
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
These labels indicate the change from an affirmative sentence to a negated sentence and vice versa. Only negation markers are annotated with this label. Negation markers appearing in the simplification are annotated with Affirmation to negation, while negation markers disappearing from the original sentence are annotated with Negation to affirmation. Example :
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
Split indicates that the original sentence is split into multiple sentences in the simplified version. For example, if the original sentence is The cat is tall and also blue and the simplified sentence The cat is tall. It is also blue., the full stop between the two simplified sentences is annotated with Split. The segment It is in It is also blue. is annotated as insert-cst (see examples). The label merge -two sentences or more grouped into one sentence -is present for reasons of coherence. However, there normally is no case where this label is applicable, as sentences are simplified one by one in ASSET. Examples:
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
It can happen that certain transformations can be annotated in more than one way, with no rules of priority taking absolute precedence. As it is impossible to cover all possibilities, an arbitrary rule is to annotate as close to the token-level as possible.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
For example, the the following sentence:
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
The transformation reports of riots → riot reports can be annotated in two ways:
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• Annotate the two groups with the label synonym
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
• Annotate (i) riots and riot as plural to singular, (ii) reports and reports as move, and (iii) of in the original sentence as delete-cst.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
Both approaches are valid, but in order to minimize disagreement, we apply the second choice in this case.
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation
C Average recall and precision values per type of simplification operations over the 9 annotators  Table 8: Number of annotations per operation found in the gold corpus, and average recall and precision at the token level for the 9 annotators, per operation (simplified sentences and original sentences). NA indicates operations not seen in the respective data. None has a count of NA because it is only applied when an annotator does not label a token.   
Simple English Wikipedia: A New Text Simpliﬁcation Task
The task of text simplification aims to reduce the complexity of text while maintaining the content (Chandrasekar and Srinivas, 1997;Carroll et al., 1998;Feng, 2008). In this paper, we explore the sentence simplification problem: given a sentence, the goal is to produce an equivalent sentence where the vocabulary and sentence structure are simpler.
Simple English Wikipedia: A New Text Simpliﬁcation Task
Text simplification has a number of important applications. Simplification techniques can be used to make text resources available to a broader range of readers, including children, language learners, the elderly, the hearing impaired and people with aphasia or cognitive disabilities (Carroll et al., 1998;Feng, 2008). As a preprocessing step, simplification can improve the performance of NLP tasks, including parsing, semantic role labeling, machine translation and summarization (Miwa et al., 2010;Jonnala-gadda et al., 2009;Vickrey and Koller, 2008;Chandrasekar and Srinivas, 1997). Finally, models for text simplification are similar to models for sentence compression; advances in simplification can benefit compression, which has applications in mobile devices, summarization and captioning (Knight and Marcu, 2002;McDonald, 2006;Galley and McKeown, 2007;Nomoto, 2009;Cohn and Lapata, 2009).
Simple English Wikipedia: A New Text Simpliﬁcation Task
One of the key challenges for text simplification is data availability. The small amount of simplification data currently available has prevented the application of data-driven techniques like those used in other text-to-text translation areas (Och and Ney, 2004;Chiang, 2010). Most prior techniques for text simplification have involved either hand-crafted rules (Vickrey and Koller, 2008;Feng, 2008) or learned within a very restricted rule space (Chandrasekar and Srinivas, 1997).
Simple English Wikipedia: A New Text Simpliﬁcation Task
We have generated a data set consisting of 137K aligned simplified/unsimplified sentence pairs by pairing documents, then sentences from English Wikipedia 1 with corresponding documents and sentences from Simple English Wikipedia 2 . Simple English Wikipedia contains articles aimed at children and English language learners and contains similar content to English Wikipedia but with simpler vocabulary and grammar. Figure 1 shows example sentence simplifications from the data set. Like machine translation and other text-to-text domains, text simplification involves the full range of transformation operations including deletion, rewording, reordering and insertion. a. Normal: As Isolde arrives at his side, Tristan dies with her name on his lips.
Simple English Wikipedia: A New Text Simpliﬁcation Task
Simple: As Isolde arrives at his side, Tristan dies while speaking her name. b. Normal: Alfonso Perez Munoz, usually referred to as Alfonso, is a former Spanish footballer, in the striker position. Simple: Alfonso Perez is a former Spanish football player. c. Normal: Endemic types or species are especially likely to develop on islands because of their geographical isolation. Simple: Endemic types are most likely to develop on islands because they are isolated. d. Normal: The reverse process, producing electrical energy from mechanical, energy, is accomplished by a generator or dynamo. Simple: A dynamo or an electric generator does the reverse: it changes mechanical movement into electric energy. 
Simple English Wikipedia: A New Text Simpliﬁcation Task
Wikipedia and Simple English Wikipedia have both received some recent attention as a useful resource for text simplification and the related task of text compression. Yamangil and Nelken (2008) examine the history logs of English Wikipedia to learn sentence compression rules. Yatskar et al. (2010) learn a set of candidate phrase simplification rules based on edits identified in the revision histories of both Simple English Wikipedia and English Wikipedia. However, they only provide a list of the top phrasal simplifications and do not utilize them in an endto-end simplification system. Finally, Napoles and Dredze (2010) provide an analysis of the differences between documents in English Wikipedia and Simple English Wikipedia, though they do not view the data set as a parallel corpus.
Simple English Wikipedia: A New Text Simpliﬁcation Task
Although the simplification problem shares some characteristics with the text compression problem, existing text compression data sets are small and contain a restricted set of possible transformations (often only deletion). Knight and Marcu (2002) introduced the Zipf-Davis corpus which contains 1K sentence pairs. Cohn and Lapata (2009) manually generated two parallel corpora from news stories totaling 3K sentence pairs. Finally, Nomoto (2009) generated a data set based on RSS feeds containing 2K sentence pairs.
Simple English Wikipedia: A New Text Simpliﬁcation Task
We generated a parallel simplification corpus by aligning sentences between English Wikipedia and Simple English Wikipedia. We obtained complete copies of English Wikipedia and Simple English Wikipedia in May 2010. We first paired the articles by title, then removed all article pairs where either article: contained only a single line, was flagged as a stub, was flagged as a disambiguation page or was a meta-page about Wikipedia. After pairing and filtering, 10,588 aligned, content article pairs remained (a 90% reduction from the original 110K Simple English Wikipedia articles). Throughout the rest of this paper we will refer to unsimplified text from English Wikipedia as normal and to the simplified version from Simple English Wikipedia as simple.
Simple English Wikipedia: A New Text Simpliﬁcation Task
To generate aligned sentence pairs from the aligned document pairs we followed an approach similar to those utilized in previous monolingual alignment problems (Barzilay and Elhadad, 2003;Nelken and Shieber, 2006). Paragraphs were identified based on formatting information available in the articles. Each simple paragraph was then aligned to every normal paragraph where the TF-IDF, cosine similarity was over a threshold or 0.5. We initially investigated the paragraph clustering preprocessing step in (Barzilay and Elhadad, 2003), but did not find a qualitative difference and opted for the simpler similarity-based alignment approach, which does not require manual annotation. 666
Simple English Wikipedia: A New Text Simpliﬁcation Task
For each aligned paragraph pair (i.e. a simple paragraph and one or more normal paragraphs), we then used a dynamic programming approach to find that best global sentence alignment following Barzilay and Elhadad (2003). Specifically, given n normal sentences to align to m simple sentences, we find a(n, m) using the following recurrence:
Simple English Wikipedia: A New Text Simpliﬁcation Task
where each line above corresponds to a sentence alignment operation: skip the simple sentence, skip the normal sentence, align one normal to one simple, align one normal to two simple, align two normal to one simple and align two normal to two simple. sim(i, j) is the similarity between the ith normal sentence and the jth simple sentence and was calculated using TF-IDF, cosine similarity. We set skip penalty = 0.0001 manually. Barzilay and Elhadad (2003) further discourage aligning dissimilar sentences by including a "mismatch penalty" in the similarity measure. Instead, we included a filtering step removing all sentence pairs with a normalized similarity below a threshold of 0.5. We found this approach to be more intuitive and allowed us to compare the effects of differing levels of similarity in the training set. Our choice of threshold is high enough to ensure that most alignments are correct, but low enough to allow for variation in the paired sentences. In the future, we hope to explore other similarity techniques that will pair sentences with even larger variation.
Simple English Wikipedia: A New Text Simpliﬁcation Task
From the 10K article pairs, we extracted 75K aligned paragraphs. From these, we extracted the final set of 137K aligned sentence pairs. To evaluate the quality of the aligned sentences, we asked two human evaluators to independently judge whether or not the aligned sentences were correctly aligned on a random sample of 100 sentence pairs. They then were asked to reach a consensus about correctness. 91/100 were identified as correct, though many of the remaining 9 also had some partial content overlap. We also repeated the experiment using only those sentences with a similarity above 0.75 (rather than 0.50 in the original data). This reduced the number of pairs from 137K to 90K, but the evaluators identified 98/100 as correct. The analysis throughout the rest of the section is for threshold of 0.5, though similar results were also seen for the threshold of 0.75.
Simple English Wikipedia: A New Text Simpliﬁcation Task
Although the average simple article contained approximately 40 sentences, we extracted an average of 14 aligned sentence pairs per article. Qualitatively, it is rare to find a simple article that is a direct translation of the normal article, that is, a simple article that was generated by only making sentencelevel changes to the normal document. However, there is a strong relationship between the two data sets: 27% of our aligned sentences were identical between simple and normal. We left these identical sentence pairs in our data set since not all sentences need to be simplified and it is important for any simplification algorithm to be able to handle this case.
Simple English Wikipedia: A New Text Simpliﬁcation Task
Much of the content without direct correspondence is removed during paragraph alignment. 65% of the simple paragraphs do not align to a normal paragraphs and are ignored. On top of this, within aligned paragraphs, there are a large number of sentences that do not align. Table 1 shows the proportion of the different sentence level alignment operations in our data set. On both the simple and normal sides there are many sentences that do not align.
Simple English Wikipedia: A New Text Simpliﬁcation Task
Operation % skip simple 27% skip normal 23% one normal to one simple 37% one normal to two simple 8% two normal to one simple 5% Table 1: Frequency of sentence-level alignment operations based on our learned sentence alignment. No 2-to-2 alignments were found in the data.
Simple English Wikipedia: A New Text Simpliﬁcation Task
To better understand how sentences are transformed from normal to simple sentences we learned a word alignment using GIZA++ (Och and Ney, 2003). Based on this word alignment, we calculated the percentage of sentences that included: re-wordings -a normal word is changed to a different simple word, deletions -a normal word is deleted, reorderings -non-monotonic alignment, splits -a normal words is split into multiple simple words, and merges -multiple normal words are condensed to a single simple word.   Table 2 shows the percentage of each of these phenomena occurring in the sentence pairs. All of the different operations occur frequently in the data set with rewordings being particularly prevalent.
Simple English Wikipedia: A New Text Simpliﬁcation Task
To understand the usefulness of this data we ran preliminary experiments to learn a sentence-level simplification system. We view the problem of text simplification as an English-to-English translation problem. Motivated by the importance of lexical changes, we used Moses, a phrase-based machine translation system (Och and Ney, 2004). 3 We trained Moses on 124K pairs from the data set and the n-gram language model on the simple side of this data. We trained the hyper-parameters of the loglinear model on a 500 sentence pair development set.
Simple English Wikipedia: A New Text Simpliﬁcation Task
We compared the trained system to a baseline of not doing any simplification (NONE). We evaluated the two approaches on a test set of 1300 sentence pairs. Since there is currently no standard for automatically evaluating sentence simplification, we used three different automatic measures that have been used in related domains: BLEU, which has been used extensively in machine translation (Papineni et al., 2002), and word-level F1 and simple string accuracy (SSA) which have been suggested  for text compression (Clarke and Lapata, 2006). All three of these measures have been shown to correlate with human judgements in their respective domains. Table 3 shows the results of our initial test. All differences are statistically significant at p = 0.01, measured using bootstrap resampling with 100 samples (Koehn, 2004). Although the baseline does well (recall that over a quarter of the sentence pairs in the data set are identical) the phrase-based approach does obtain a statistically significant improvement.
Simple English Wikipedia: A New Text Simpliﬁcation Task
To understand the the limits of the phrase-based model for text simplification, we generated an nbest list of the 1000 most-likely simplifications for each test sentence. We then greedily picked the simplification from this n-best list that had the highest sentence-level BLEU score based on the test examples, labeled Moses-Oracle in Table 3. The large difference between Moses and Moses-Oracle indicates possible room for improvement utilizing better parameter estimation or n-best list reranking techniques Ge and Mooney, 2006).
Simple English Wikipedia: A New Text Simpliﬁcation Task
We have described a new text simplification data set generated from aligning sentences in Simple English Wikipedia with sentences in English Wikipedia. The data set is orders of magnitude larger than any currently available for text simplification or for the related field of text compression and is publicly available. 4 We provided preliminary text simplification results using Moses, a phrase-based translation system, and saw a statistically significant improvement of 0.005 BLEU over the baseline of no simplification and showed that further improvement of up to 0.034 BLEU may be possible based on the oracle results. In the future, we hope to explore alignment techniques more tailored to simplification as well as applications of this data to text simplification.
The Role of Text Simplification Operations in Evaluation
TS methods transform complex text fragments into their simple variants, according to specific operations and audiences. Non-native speakers can significantly benefit from the substitution of complex to simple words [1], while other audiences, such as people with aphasia, will benefit more from short, simple sentences [2]. Although, categorising what complexity means for different audiences is useful for evaluation, TS remains a challenging task to benchmark for the following reasons: 1) The basic concept of simplicity (relying on language complexity) is vague and hard to define quantitatively, which means that proficient language users usually come up with different simplifications for a given sentence; 2) The possible usages of TS include scenarios aimed at different target audiences (e.g., children, non-native readers, people with aphasia or dyslexia) and domains (e.g., scientific texts, medical and legal documents), who may require different simplification methods; and 3) Using a gold-standard for TS evaluation requires human annotations which is time consuming and costly. This is usually avoided in a way similar to other Natural Language Generation (NLG) tasks (e.g., machine translation) by obtaining human annotated reference simplifications and evaluating systems based on their similarities to these. Although this mechanism of evaluation allows an unlimited number of systems and variants to be evaluated without further human effort, there are a number of factors we have to consider when interpreting the results.
The Role of Text Simplification Operations in Evaluation
Firstly, there may be many equally good simplifications for a given sentence, so comparison to a single reference may be penalising them unfairly. Despite the existing multiple references in some TS datasets, these cannot capture the rich diversity possible in simplification. Secondly, automatic similarity measures, such as BLEU [3], ROUGE [4] or SARI [5] have been previously shown to have limitations (e.g., weak correlation with human judgement, dependency on quality references, failure to capture task-dependent aspects such as simplicity), both in general tasks [6,7] and in the context of TS [5,8,9]. Thirdly, how data is split between training and test data can influence results. This is well-known in general [10], but has not attracted much attention in TS. Finally, simplification operations may be unevenly distributed in TS datasets, affecting the types of simplifications that a model learns to produce. Test splits may not reflect the same simplification operations as in the training split from the same dataset.
The Role of Text Simplification Operations in Evaluation
In this paper, we explore the impact of data splits (random and stratified) on English TS datasets and set up a systematic benchmark on the existing datasets with altered distributions. Our contributions are: 1) An operations-based analysis of TS datasets generated by stratification algorithms; 2) A performance evaluation on experimental operation-based datasets; and 3) Recommendations towards a standardised practice for building and evaluating new TS datasets.
The Role of Text Simplification Operations in Evaluation
We conducted a systematic analysis of the key operations we have identified for all commonly available TS datasets. Initially, we analysed the amount of deletions, insertions and replacements for the different subsets of each TS dataset (i.e., train, development and test when available). We did not include the split operation, since our preliminary analysis using HSplit did not show relevant changes from an edit-distance perspective. Next, we analysed the impact of these operations on the output sentences, comparing how much a complex sentence is changed with the presence of these transformations (Section 3.1 and Section 3.2). Furthermore, we also analysed their distribution, with regards to these simplification operations, proposing new scenarios to benchmark on these new distributions (Section 3.3).
The Role of Text Simplification Operations in Evaluation
We performed our analysis using common Wikipedia-based TS datasets, including: WikiSmall and WikiLarge [11], TurkCorpus [5] and ASSET [13] 1 . In particular, we focused on analysing the original TS datasets and our proposed experimental datasets, which are modified versions of WikiLarge and WikiSmall using different distribution methods. We have chosen these resources, since they provide a test, a development and training subset, which are essential for our distribution experiments. We analysed these datasets under the following classifications:
The Role of Text Simplification Operations in Evaluation
Original distribution: we examined all subsets of the original TS datasets with no modification by applying the metrics defined in section 3.2. We quantified the distribution divergence between subsets (test compared to train and test compared to development) by calculating the Kullback-Lieber (KL-divergence) [21] and Jensen-Shannon divergence (JSD) [22]. As a result, the Wikilarge dataset had a KL-divergence of 0.46 and a JSD divergence of 0.41, confirming that the split of this dataset is not truly random. This can be compared in detail by observing the distribution of these subsets in Figure 1a. Also, we determined that there is a significant amount of sentences with no operations and sentences that have changed 100% during the simplification process. By performing a post-hoc manual inspection of these cases, we noticed that these corresponded to inaccurate simplifications as a product of bad alignments (i.e., poor alignments or noise). Given these results, we proposed additional distributions to improve the distribution of simplification operations in WikiSmall and WikiLarge datasets.
The Role of Text Simplification Operations in Evaluation
Random distribution: to create randomly distributed datasets, we merged all the subsets from the original dataset into a single dataset, shuffled data using Numpy [23] and recreated the subsets keeping their original size. We repeated this process by using 5 different random seeds (155, 324, 393, 728, 989). The seeds selection was randomly generated, except for 324, which belongs to the original implementation of EditNTS and to the initial explorations in our previous work [20]. In Figure 1, we can see a comparison of the original (Figure 1a) and the random distribution ( Figure 1c) for seed 324 2 .
The Role of Text Simplification Operations in Evaluation
Minimised poor-alignments distribution: we manually inspected sentences shown at the right-most of Figure 1a and we observed that sentences close to 100% of change correspond to incorrect simplifications or alignments. Based on this, we created new datasets by removing these poor-alignments ranging from 2% to 20% of sentences with the worst alignment from the original dataset. These splits were not randomised to isolate the effect of removing the poor alignments in TS datasets and duplicates were removed. Figure 1d and 1e show the decrease in the percentage of change in WikiLarge by using this heuristic, including a significantly higher reduction of change in the tests sets compared to the other subsets.
The Role of Text Simplification Operations in Evaluation
Stratified distribution: sentences in TS datasets can be analysed not only by the changes done from the original to the simplified sentence, but also by the operation type. Our main goal for building new stratified splits is to have similar number of operations of each type (e.g., deletes, inserts and replacements) in each subset. Since a single sentence simplification can involve multiple operations, it is difficult to have the desired distribution between subsets. Among the algorithms evaluated, we selected Monte Carlo Algorithm 3 as our best approach based on the operations distribution. The original datasets were distributed according to this algorithm; datasets subsets were rebuilt and then analysed, likewise to the random distribution. We generated 500,000 random splits searching for one with the best standard deviation between the amount of DELETE, INSERT and REPLACE in each subset. At every 100,000 iterations, we saved the 2 best candidates based on their best standard deviation: one in the training set and one in the development and test sets, minimising the difference in their individual standard deviation. For WikiLarge, the more suitable splits were iteration 200,000 and 400,000, whereas for WikiSmall these were iterations 300,000 and 500,000. We show the latter in the Appendix.
The Role of Text Simplification Operations in Evaluation
Once the original and new experimental datasets were created and analysed, we evaluated the effect they had on the performance of the EditNTS [18] model by measuring the change in SARI score when training on the redistributed datasets. We adapted the original code with some minor modifications to run in our setting, including: model randomisation with fixed seeds, scripts for data preprocessing and the automation of test sets evaluation. We trained the models on the original and the experimental subsets (poor-alignments reduction, random and stratified distributions) using the same hyper parameters from the EditNTS model (batch size, epochs, dropout, and learning rate). Next, we evaluated the performance of the newly trained model by using ASSET as an external test subset. Finally, we manually inspected a sample of the model outputs for all the proposed datasets. The adaptations for the EditNTS model, the experimental subsets, the model outputs and the source code are documented via GitHub 4 . 
The Role of Text Simplification Operations in Evaluation
Wikipedia-based TS datasets were created collaboratively by volunteers, with the main goal to support learning for non-native speakers. In addition to the rule of writing in Simple English 5 , there were no specific guidelines on how to simplify text, such as the type and the amount of simplifications allowed, or whether it should match the original Wikipedia article.
The Role of Text Simplification Operations in Evaluation
Except for specific studies done by Alva-Manchego et al. [13] and Xu et al. [12], there is no accurate notion of what simplification operations are performed in all TS datasets. These studies are less comprehensive since they investigate specific TS datasets or limited TS operations. Consequently, we analysed common TS datasets by using the following metrics: Simplification operations count: we quantified the percentage of edits required to transform a complex sentence to a simple one (henceforth, edit-distance [24]). To achieve this, we calculated the edit-distance between two sentences by adapting the Wagner-Fischer algorithm [25] to determine changes from characters-level to a token-level (e.g., words). This method defines how many tokens in the complex sentence were changed in the simplified output (e.g., 2 tokens that were deleted from one version to another is equivalent to 2 changes). Prior to the analysis, sentences were changed to lowercase. Values are expressed as a change percentage, where 0% indicates sentences with no changes and 100% indicates completely different sentences. In Figure 1 we show the edit-distance analysis for WikiLarge, for the original splits ( Figure 1a) and also, for the randomised (Figure 1c), poor-alignments-based (Figure 1d, 1e) and stratified splits (Figure 1f). Random and stratified experimental splits clearly show a more even distribution of sentences between subsets, according to the amount of change required to obtain a new simplification from a complex sentence. On the other hand, removing poor-alignments, without a proper distribution leaves the tests sets with the majority of samples with minimal or no change.
The Role of Text Simplification Operations in Evaluation
Simplification operations types: after extracting the token-level edits done between two sentences, we classified them into simplification operations: INSERT (a token(s) has been added), DELETE (a token(s) has been removed) and REPLACE (a token(s) has been substituted). These three basic operations can be performed at a lexical-level 6 . We show in Figure 1b the simplification operations types for WikiLarge dataset. These results not only show how unbalanced these operations are between subsets but also the predominance of DELETE operations in the WikiLarge dataset for the development and training subsets. Also, the DELETE effect is also noticeable when we manually checked the outputs of the models. A majority of the simplification operations performed deletions in the original sentence, rather than, performing substitutions or insertions. Furthermore, in Figure 3 we performed a more exhaustive comparison, analysing the operations count and their distribution in all our experiments.
The Role of Text Simplification Operations in Evaluation
We performed an evaluation of the proposed datasets by retraining the EditNTS model with development and training subsets (for both WikiSmall and WikiLarge) 7 . Once models were trained, we evaluated their performance using the SARI scores provided by the model evaluation scripts. In our evaluation setting, the test subset of the ASSET dataset was used to test the trained models. Also, we reported the average results for the evaluation of all ASSET references in each complex sentence, since our implementation based on EditNTS model evaluated one  test reference at a time. Figure 2 shows a comparison between SARI scores in all the models, for WikiLarge-based and WikiSmall-based models. We also include the error bars (standard deviation of ASSET averaged observations) for all calculations. We observed that randomising the distribution and reducing the poor alignments helped for the WikiSmall dataset. Meanwhile, using the Monte Carlo algorithm and performing more substantial reductions in the distribution had a better contribution for WikiLarge.
The Role of Text Simplification Operations in Evaluation
Firstly, we conclude from our analyses (Section 3) that the original TS datasets do not follow an even distribution between subsets. We observe that the test, development and train subsets are different when measured by the amount of change from the simple to the complex sentence. Furthermore, our evaluation on these experimental subsets show that the random distributions provide significant variations in SARI scores, even though its composition is similar. For Wik-iSmall (Figure 2a), the results between random splits showed an increase of up to 7 percentage points in SARI score, just by randomising dataset composition and rebuilding the dataset. For WikiLarge (Figure 2b), we found a similar effect with the Monte Carlo algorithm, which is a split randomised by 200,000 iterations. The main difference is that this algorithm selects the best score among all the generated random samples, rather than any of them. In this setting, the variation of the SARI score is about 5 percentage points. The difference in SARI score should be interpreted as a measure of simplicity gain, which provides a relative comparison of correctness between simplifications. However, it cannot be interpreted as the best possible simplification, since this evaluation metric fails to measure simplicity alone, as mentioned in Section 2.
The Role of Text Simplification Operations in Evaluation
Secondly, WikiSmall and WikiLarge datasets show a significant amount of noise and sentences that are not simplifications. Interestingly, we can see in Figure 1e that when we aggressively removed 15% of the dataset, it reduced considerably the amount of sentences with a percentage of change higher than 40%. Despite this, the performance between the original model orig_100% and its reduced version orig_85% did not change more than 0.02% in both WikiSmall and WikiLarge. For the model orig_80% (which has 20% of estimated noise reduction), we observed a different scenario in WikiSmall; since, in comparison with orig_100% model the performance of this dataset dropped 2.6%. WikiSmall dataset is significantly smaller than WikiLarge (3X), and so, such a reduction affects a higher number of real simplifications. In contrast, Wikilarge model orig_98% has a minimal number of noise reduction, keeping its composition almost unchanged. We presume that the decrease in the model performance relates to having the same dataset composition but with less sentence samples (despite their lower quality).
The Role of Text Simplification Operations in Evaluation
Thirdly, we discuss the datasets composition with respect to the operations count ( Figure 3). Due to the large size of the training corpus, the count in the train subset is similar for all the datasets. However, that is not the case of the test and the development subsets, where we noticed meaningful differences. We observe a consistent decrease in all the operations for the models where we removed the 'poor-alignments'. Nevertheless, as we mentioned earlier, orig_80% was the only model which presented a decrease in performance, with a minimal amount of edit operations. On the contrary, despite the similar distribution in operations between random datasets we did observe performance variations between these models. It is relevant to consider that the test and development subsets are quite smaller than the training subset (359 test / 992 dev / 296,402 train in the original sentence pairs). We presume that this could minimise the effect of new distributions towards the model performance. As future work, we would consider changing the original subset sizes to explore further the effect of simplification operations.
The Role of Text Simplification Operations in Evaluation
Although the evaluation metrics and model outputs are not globally providing enough information about a dataset, we believe it is important to follow a structured setting to value the quality of a dataset. To ensure interpretable methods for dataset quality assessment, we make the following recommendations for TS dataset evaluation.
The Role of Text Simplification Operations in Evaluation
Noisy alignments detection: current TS datasets are automatically aligned, hence, these are likely to have incorrect or unaligned sentence pairs. We propose a heuristic in which these inaccurate alignments can be detected by quantifying the amount of change between the complex sentence and the gold-reference ones. This can be implemented by sorting TS datasets using edit-distance values so sentences with higher amount of changes are grouped together, providing a straight-forward way for detection and removal of noise. The ideal threshold in which sentences are removed can be determined by visually inspecting these groups.
The Role of Text Simplification Operations in Evaluation
Simplification operations distribution: depending on the audience, some simplification operations can be more useful than others. Ideally, we would expect not only a variety of simplification operations but also, a similar distribution of operations between subsets tailored to a given simplification need. There are valid scenarios in which particular operations could be enough (e.g., REPLACE operation for complex word simplification for non-native speakers). Other areas such as news simplification, require more elaborate constructions which not only involves simplifications at a lexical level, but also at a discourse level (e.g, news for general public targeted for children at schools in the Newsela dataset). By using token-based edit distance, we can perform a global count of simplification operations performed and an evaluation of their distribution as an aid for stratifying TS datasets as needed.
The Role of Text Simplification Operations in Evaluation
Datasets stability: from our experiments, we have observed that dataset distribution significantly affects TS model performance (measured by an increase or decrease in SARI score). Our recommendation is to perform a dataset randomisation with different random seeds to evaluate the impact of data distribution in TS models performance. In addition, datasets of significant size, such as Wikilarge, showed to be more stable in this setting (less variation in SARI score between random seeds).
The Role of Text Simplification Operations in Evaluation
In this paper, we have performed a systematic analysis of the most common TS operations demonstrating the statistical limitations of English TS datasets. Our analysis can be reproduced through our published scripts, which can also be used to analyse any other TS parallel dataset for quality assessment. Moreover, we carried out a detailed evaluation of all of our experimental settings, including distributions with poor-alignments reduction, randomisation and stratification using the Monte Carlo algorithm. Finally, we have proposed a set of recommendations for the creation of more reliable and standardised datasets for a better environment of TS evaluation resources. 