\section{Introduction}

To search relevant research papers for a particular field is a daily core activity of researchers.
Scientists usually pose queries to academic search engines and skim through the text of the found articles to assess their relevance.
However, academic search engines cannot assist researchers adequately in these tasks since most research papers are plain PDF files and not machine-interpretable \citep{Brack2020RequirementsAnalysisORKG,Safder2019,Xiong2017ExplicitSR}.
The exploding number of published articles aggravates this situation further~\citep{bornmann15growth}.
Therefore, scholarly literature mining with natural language processing (NLP) gains more and more relevance in the communities of information retrieval and scientometrics \citep{Cabanac2020,Nasar2018InformationEF,Safder2019}.
\begin{comment}
\begin{figure}[t]
\vspace{2em}
\small
\centering%
\begin{tabular}{| p{\dimexpr0.99\textwidth-2\tabcolsep-\arrayrulewidth\relax}|   }
\hline
\\
Transfer learning is aimed to make use of valuable knowledge in a source domain to help model performance in a target domain. It is particularly important to neural networks, which are very likely to be overfitting. In some fields like image processing, many studies have shown the effectiveness of neural network-based transfer learning. For neural NLP, however, existing studies have only casually applied transfer learning, and conclusions are inconsistent. \hlcyan{In this paper, we conduct systematic case studies and provide an illuminating picture on the transferability of neural networks in NLP}.
\\
\\
\hline
\end{tabular}
\caption{Example of an abstract taken from \citet{Mou2016HowTA}, in which the sentence describing the \emph{objective} of the paper is coloured. Highlighting relevant sentences in a paper can support researchers explore literature more efficiently.}
\label{fig:highlight-example}
\vspace{-1em}
\end{figure}
\end{comment}

\begin{comment}
\begin{figure}[t]
\vspace{2em}
\footnotesize
\centering%
\begin{tabular}{| p{\dimexpr0.99\textwidth-2\tabcolsep-\arrayrulewidth\relax}|   }
\hline
\\
\hlcyan{Deep Learning's recent successes have mostly relied on Convolutional Networks, which exploit fundamental statistical properties of images, sounds and video data: the local stationarity and multi-scale compositional structure, that allows expressing long range interactions in terms of shorter, localized interactions. However, there exist other important examples, such as text documents or bioinformatic data, that may lack some or all of these strong statistical regularities.} \hl{In this paper we consider the general question of how to construct deep architectures with small learning complexity on general non-Euclidean domains, which are typically unknown and need to be estimated from the data.} \hlblue{In particular, we develop an extension of Spectral Networks which incorporates a Graph Estimation procedure, that we test on large-scale classification problems, matching or improving over Dropout Networks with far less parameters to estimate.}\\
\\
\hline
\end{tabular}
\caption{Example of an annotated abstract taken from the CSABSTRUCT dataset \citep{Cohan2019PretrainedLM}, in which sentences describing the \emph{background} (green), \emph{objective} (yellow), and \emph{methods} (magenta) of the paper are coloured. Highlighting relevant sentences in a paper can support researchers explore literature more efficiently.}
\label{fig:highlight-example}
\vspace{-1em}
\end{figure}
\end{comment}

\begin{figure}[t]
\vspace{2em}
\footnotesize
\centering%
\begin{tabular}{| p{\dimexpr0.99\textwidth-2\tabcolsep-\arrayrulewidth\relax}|   }
\hline
\\
\hlcyan{Gamification has the potential to improve the quality of learning by better engaging students with learning activities.} \hl{Our objective in this study is to evaluate a gamified learning activity along the dimensions of learning, engagement, and enjoyment.} \hlblue{The activity made use of a gamified multiple choice quiz implemented as a software tool and was trialled in three undergraduate IT-related courses. A questionnaire survey was used to collect data to gauge levels of learning, engagement, and enjoyment.} \hlpurple{Results show that there was some degree of engagement and enjoyment. The majority of participants (77.63 per cent) reported that they were engaged enough to want to complete the quiz and 46.05 per cent stated they were happy while playing the quiz...}\\
\\
\hline
\end{tabular}
\caption{An annotated abstract taken from the CSABSTRUCT dataset \citep{Cohan2019PretrainedLM}, in which sentences describing the \emph{background} (green), \emph{objective} (yellow), \emph{methods} (magenta), and \emph{results} (cyan) of the paper are coloured.}
\label{fig:highlight-example}
\vspace{-1em}
\end{figure}

The task of \emph{sequential sentence classification} enables classifying sentences in a research paper in categories like \textit{objective}, \textit{methods}, or \textit{results} \citep{Dernoncourt2017PubMed2R}. Figure \ref{fig:highlight-example} shows an example of an abstract with classified sentences.
Such a semantification of sentences can help algorithms focus on relevant elements of text and thus assist information retrieval systems \citep{Neves2019EvaluationOS,Safder2019} or other downstream tasks such as scientific concept extraction or trend analysis.
The task is called \emph{sequential} to distinguish it from the general \emph{sentence classification} task where a sentence is classified in isolation, i.e. without using local context.
However, in research papers the meaning of a sentence is often informed by the context from neighbouring sentences,
e.g. sentences describing the methods usually precede sentences describing the results.

In previous work, several approaches have been proposed for \emph{sequential sentence classification} (e.g. \citep{Asadi2019AutomaticZI,Jin2018HierarchicalNN,yamada-etal-2020-sequential}), and several datasets were annotated for various scientific domains (e.g. \citep{Dernoncourt2017PubMed2R,Fisas2015OnTD,Goncalves20,SteadSBV19}). The datasets contain either abstracts or full papers and were annotated with domain-specific sentence classes. Although this task is well studied, several research questions remain open.

First, the field lacks studies on transfer learning across different scientific domains and especially for \emph{sequential} sentence classification.
Transfer learning enables the combination of knowledge from multiple datasets to improve classification performance and thus to reduce annotation costs.
The annotation of scientific text is particularly costly since it demands expertise in the article’s domain~\citep{augenstein2017semeval,Brack2020DomainindependentEO,gabor2018semeval}.
However, studies on transfer learning revealed that the success of transferring neural models depends largely on the relatedness of the tasks and transfer learning with unrelated tasks may even degrade the performance \citep{Mou2016HowTA,Ruder2019Neural,Pan10,Semwal2018}.
Two tasks are related if there exists some implicit or explicit relationship between the feature spaces \citep{Pan10}.
On the other hand, every scientific domain is characterised by its specific terminology and phrasing, which yields different feature spaces. Thus, it is not clear to which extent datasets from different scientific disciplines are related.
This raises the following research questions (RQ) for the task of sequential sentence classification:
\begin{itemize}
\item[] RQ1: To which extent are datasets from different scientific domains related?
\item[] RQ2: Which transfer learning approach does work best?
\item[] RQ3: Which neural network layers are transferable under which constraints?
\item[] RQ4: Is it beneficial to train a multi-task model with multiple datasets?
\end{itemize}

Normally, every dataset has a domain-specific annotation scheme that consists of a set of associated sentence classes. This raises the second set of research questions with regard to the consolidation of these annotation schemes. Prior work \citep{Liakata2010CorporaFT} annotated a dataset multiple times with different schemes, and analysed the multivariate frequency distributions of the classes. They found that the investigated schemes are complementary and should be combined.
However, annotating datasets multiple times is costly.
To support the consolidation of different annotation schemes across domains, we examine the following research questions:
\begin{itemize}
\item[] RQ5: Can a model trained with multiple datasets recognise the semantic relatedness of classes from different annotation schemes?
\item[] RQ6: Can we derive a consolidated, domain-independent annotation scheme to compile a new dataset to train a domain-independent classifier?
\end{itemize}

Finally, current approaches for sequential sentence classification are designed either for abstracts or full papers only.
One reason is that these text types follow rather different structures. For instance, in abstracts different sentence classes directly follow one another, while in full papers there is often no sentence class change in larger paragraphs and text parts.
Typically, deep learning is used for abstracts \citep{Cohan2019PretrainedLM,Goncalves20,Dernoncourt2016NeuralNF,Jin2018HierarchicalNN,yamada-etal-2020-sequential} since presumably more training data is available, whereas for full papers, also called \emph{zone identification}, still hand-crafted features and linear models have been suggested \citep{Asadi2019AutomaticZI,Badie2018ZoneIB,Fisas2015OnTD,liakata2012automatic}.
However, deep learning approaches have also been employed successfully on full papers in related tasks such as argumentation mining \citep{lauscher2018a}, scientific document summarisation~\citep{AbuRaed2020,cohan-etal-2018-discourse,ghosh-roy-etal-2020-summaformers}, or n-ary relation extraction \citep{Jia2019}.
Thus, the potential of deep learning has not been fully exploited yet for sequential sentence classification on full papers, and no unified solution for abstracts as well as full papers exists. This raises the research question:
\begin{itemize}
\item[] RQ7: Can a unified deep learning approach be applied to text types with very different structures like abstracts or full papers?
\end{itemize}

In this paper, we investigate these research questions and present the following contributions:
(1)~We introduce a novel multi-task learning framework for sequential sentence classification.
Comprehensive experimental results demonstrate that our multi-task learning approach successfully makes use of datasets from different scientific domains, with different annotation schemes, that contain abstracts or full papers.
In particular, we outperform state-of-the-art approaches for full paper datasets significantly, while obtaining slightly better or equivalent results for datasets consisting of abstracts.
(2)~Furthermore, we propose and evaluate an approach to semi-automatically identify semantically related classes from different annotation schemes and present an analysis of four annotation schemes.
Based on the analysis, we suggest a domain-independent annotation scheme and compile a new dataset that enables to classify sentences in a domain-independent manner.
(3)~Our proposed unified deep learning approach can handle both text types, abstracts or full papers, which differ noticeably in their structure.
(4)~To facilitate further research, we make our source code publicly available: \url{https://github.com/arthurbra/sequential-sentence-classification}.

The remainder of the paper is organised as follows: Section 2 summarises related work on sentence classification in research papers and transfer learning in NLP. Our proposed approaches are presented in Section 3. The setup and results of our experimental evaluation are reported in Section 4 and 5, while Section 6 concludes the paper and outlines areas of future work.

\section{Related Work}
\label{sec:related_work}
This section outlines datasets for sentence classification in scientific texts and describes machine learning methods for this task.
Furthermore, we briefly review transfer learning methods.
For a more comprehensive overview about information extraction from scientific text, we refer to \citet{Nasar2018InformationEF}.

\subsection{Sequential Sentence Classification in Scientific Text}
\label{sec:ml_approaches}

\paragraph{Datasets:}
As depicted in Table~\ref{tab:comparison_sentence_classification}, various annotated benchmark datasets for sentence classification in research papers come from several domains, e.g. PubMed-20k~\citep{Dernoncourt2017PubMed2R} consists of biomedical randomised controlled trials, NICTA-PIBOSO~\citep{Kim2011AutomaticCO} comes from evidence-based medicine, Dr. Inventor dataset~\citep{Fisas2015OnTD} from computer graphics, and the ART/Core Scientific Concepts (CoreSC) dataset \citep{Liakata2010CorporaFT} from chemistry and biochemistry. Most datasets cover only abstracts, while ART/CoreSC and Dr. Inventor cover full papers. Furthermore, each dataset has five to 11 different sentence classes, which are more domain-independent (e.g. Background, Methods, Results, Conclusions) or more domain-specific (e.g. Intervention, Population \citep{Kim2011AutomaticCO}, or Hypothesis, Model, Experiment \citep{Liakata2010CorporaFT}).

\begin{comment}
Available annotated benchmark datasets come from several domains, e.g. PubMed-20k~\citep{Dernoncourt2017PubMed2R} consists of biomedical randomised controlled trials, NICTA-PIBOSO~\citep{Kim2011AutomaticCO} comes from evidence-based medicine, Dr. Inventor dataset~\citep{Fisas2015OnTD,lauscher-etal-2018-argument} from computer graphics, and the ART/Core Scientific Concepts (CoreSC) dataset \citep{Liakata2010CorporaFT} from chemistry and biochemistry
Multi-label Argumentative Zoning for English Abstracts
(MA\-ZEA)~\citep{Dayrell2012RhetoricalMD} from physics, engineering, and life and health sciences,
Emerald~100k from management, information science and engineering disciplines~\citep{SteadSBV19},
as well as
CSABSTRUCT~\citep{Cohan2019PretrainedLM}
and CS-Abstracts~\citep{Goncalves20} from computer science.
The datasets PubMed-20k, NICTA-PIBOSO, MAZEA, Emerald~100k, CSABS\-TRUCT and CS-Abstracts cover only abstracts, while the other ones cover full papers.
Furthermore, each dataset has five to 11 different sentence classes, which are more domain-independent (e.g. Background, Methods, Results, Conclusions) or more domain-specific (e.g. Intervention, Population \citep{Kim2011AutomaticCO}, or Hypothesis, Model, Experiment \citep{Liakata2010CorporaFT}).

\end{comment}

\begin{sidewaystable*}%
\footnotesize
\caption{Characteristics of benchmark datasets for sentence classification in research papers.}
\label{tab:comparison_sentence_classification}
\begin{tabular}{l|l|r|l|l}
\textbf{Dataset}               & \textbf{Domains}                                                                                & \textbf{\# Papers} & \textbf{Text Type}  & \textbf{Sentence Classes}    \\ \hline
PubMed-20k     \citep{Dernoncourt2017PubMed2R} & Biomedicine                                                                              & 20,000           & abstracts & \begin{tabular}[c]{@{}l@{}}Background\\ Objective\\ Methods\\ Results, Conclusion\end{tabular} \\ \hline
NICTA-PIBOSO   \citep{Kim2011AutomaticCO}      & Biomedicine                                                                              & 1,000            & abstracts & \begin{tabular}[c]{@{}l@{}}Background\\ Intervention\\ Study\\ Population\\ Outcome, Other\end{tabular} \\ \hline
CSABSTRUCT     \citep{Cohan2019PretrainedLM}   & Computer Science                                                                         & 2,189            & abstracts & \begin{tabular}[c]{@{}l@{}}Background\\ Objective\\ Method\\ Result, Other\end{tabular} \\ \hline
CS-Abstracts   \citep{Goncalves20}             & Computer Science                                                                         & 654              & abstracts & \begin{tabular}[c]{@{}l@{}}Background\\ Objective\\ Methods\\ Results, Conclusions\end{tabular} \\ \hline
Emerald 100k   \citep{SteadSBV19}              & \begin{tabular}[c]{@{}l@{}}Management\\ Information Science\\ Engineering\end{tabular}   & 103,457          & abstracts & \begin{tabular}[c]{@{}l@{}}Purpose\\ Design/methodology/approach\\ Findings\\ Originality/value\\ Social implications\\ Practical implications\\ Research limitations/implications\end{tabular} \\ \hline
MAZEA          \citep{Dayrell2012RhetoricalMD} & \begin{tabular}[c]{@{}l@{}}Physics\\ Engineering\\ Life and Health Sciences\end{tabular} & 1,335            & abstracts & \begin{tabular}[c]{@{}l@{}}Background\\ Gap, Purpose\\ Method\\ Result, Conclusion\end{tabular} \\ \hline
Dr. Inventor   \citep{Fisas2015OnTD}           & Computer Graphics                                                                        & 40               & full paper & \begin{tabular}[c]{@{}l@{}}Background\\ Challenge\\ Approach\\ Outcome, Future Work\end{tabular} \\ \hline
ART/CoreSC     \citep{Liakata2010CorporaFT}    & \begin{tabular}[c]{@{}l@{}}Chemistry\\ Computational Linguistic\end{tabular}             & 225              & full paper & \begin{tabular}[c]{@{}l@{}}Background\\ Motivation, Goal\\ Hypothesis\\ Object\\ Model, Method\\ Experiment, Result\\ Observation, Conclusion\end{tabular}
\end{tabular}
\end{sidewaystable*}

\paragraph{Approaches for Abstracts:}
\label{sec:approaches_abstract}
Deep learning has been the preferred approach for sentence classification in  abstracts in recent years~\citep{Cohan2019PretrainedLM,Goncalves20,Dernoncourt2016NeuralNF,Jin2018HierarchicalNN,yamada-etal-2020-sequential}. These approaches follow a common \emph{hierarchical sequence labelling architecture}: (1) a word embedding layer encodes tokens of a sentence to word embeddings, (2) a sentence encoder transforms the word embeddings of a sentence to a sentence representation, (3) a context enrichment layer enriches all sentence representations of the abstract with context from surrounding sentences, and (4) an output layer predicts the label sequence.

As depicted in Table~\ref{tab:comparison_approaches_abstracts}, the approaches vary in different implementations of the layers.
For instance, \citet{Jin2018HierarchicalNN} use Word2Vec~\citep{Mikolov13Word2Vec} pre-trained on biological text as word embeddings, whereas \citet{Cohan2019PretrainedLM} use SciBERT~\citep{Beltagy2019SciBERTPC}, that is Bidirectional Encoder Representations from Transformers (BERT) \citep{Devlin2018BERTPO} pre-trained on scientific text, \citet{yamada-etal-2020-sequential} apply BERT pre-trained on biological text,
and \citet{Goncalves20} employ Global Vectors (GloVe) \citep{Pennington2014GloveGV}.
For sentence encoding, \citet{Dernoncourt2016NeuralNF} and \citet{Jin2018HierarchicalNN} contextualise word embeddings via a bidirectional long-short-term-memory (Bi-LSTM)~\citep{Hochreiter1997LongSM}, and use the output of the Bi-LSTM or attention pooling to form a sentence representation vector. \citet{Goncalves20} employ a Convolutional Neural Network (CNN) with max-pooling as a sentence encoder, while
\citet{yamada-etal-2020-sequential} use the classification token ([CLS]) of BERT.
A Bi-LSTM or bidirectional gated recurrent unit (Bi-GRU)~\citep{ChoMGBBSB14} is used for the context enrichment layer.
To predict the label sequence, a Conditional Random Field (CRF) \citep{Lafferty2001ConditionalRF} is used to capture the interdependence between labels (e.g. \emph{Results} usually follow \emph{Methods}). \citet{yamada-etal-2020-sequential} form spans of sentence representations and Semi-Markov CRFs to predict the label sequence by considering all possible span sequences of various lengths. Thus, their approach can better label longer continuous sentences, but is computationally more expensive than a CRF.
\citet{Cohan2019PretrainedLM} obtain contextual sentence representations directly by fine-tuning SciBERT.
All sentences separated by the separation token ([SEP]) are processed by Sci\-BERT and the output vector corresponding to the separation token is used as a sentence representation. However, their approach can process only about 10 sentences at once since BERT supports sequences of up to 512 tokens only. Thus, long text has to be split into multiple chunks so that long-range dependencies between sentences may be lost.

\begin{table*}[tb]
\caption{Comparison of deep learning approaches for sequential sentence classification in abstracts.}
\label{tab:comparison_approaches_abstracts}
\begin{tabular}{l|l|l|l|l}
\textbf{Approach}                                                       & \begin{tabular}[c]{@{}l@{}}\textbf{Word} \\ \textbf{embedding}\end{tabular}        & \begin{tabular}[c]{@{}l@{}}\textbf{Sentence} \\  \textbf{encoding}\end{tabular}              & \begin{tabular}[c]{@{}l@{}}\textbf{Context} \\ \textbf{enrichment}\end{tabular} & \begin{tabular}[c]{@{}l@{}}\textbf{Output} \\ \textbf{layer}\end{tabular}    \\ \hline
\begin{tabular}[c]{@{}l@{}}Dernoncourt \\ and Lee (2016)\end{tabular} & \begin{tabular}[c]{@{}l@{}}Character Emb.\\ + GloVe\end{tabular} & \begin{tabular}[c]{@{}l@{}}Bi-LSTM with\\ concatenation\end{tabular}                                                                  & -                                                             & CRF                                                        \\ \hline
\begin{tabular}[c]{@{}l@{}}Jin and \\ Szolovits (2018)\end{tabular}   & Bio word2vec                                                     & \begin{tabular}[c]{@{}l@{}}Bi-LSTM with \\ attention pooling\end{tabular} & Bi-LSTM                                                       & CRF                                                        \\ \hline
\begin{tabular}[c]{@{}l@{}}Cohan et al.\\ (2019)\end{tabular}                & SciBERT                                                          & SciBERT-[SEP]                                                             & SciBERT-[SEP]                                                 & softmax                                                    \\ \hline
\begin{tabular}[c]{@{}l@{}}Gon{\c{c}}alves et al.\\ (2020)\end{tabular}     & GloVe                                                            & \begin{tabular}[c]{@{}l@{}}CNN with \\ max pooling\end{tabular}           & Bi-GRU                                                        & softmax                                                    \\ \hline
\begin{tabular}[c]{@{}l@{}}Yamada et al.\\ (2020)\end{tabular}              & \begin{tabular}[c]{@{}l@{}}BERT from \\ PubMed\end{tabular}      & BERT-[CLS]                                                                & Bi-LSTM                                                       & \begin{tabular}[c]{@{}l@{}}Semi-Markov \\ CRF\end{tabular}
\end{tabular}
\end{table*}

\paragraph{Approaches for Full Papers:}
For full papers, logistic regression, support vector machines and CRFs with hand-crafted features have been proposed \citep{Asadi2019AutomaticZI,Badie2018ZoneIB,Fisas2015OnTD,liakata2012automatic,teufel1999argumentative,teufel2009towards}.
They represent a sentence with various syntactic and linguistic features, such as n-grams, part-of-speech tags, or citation markers, which were engineered for the respective datasets.
\citet{Asadi2019AutomaticZI} also exploit semantic features obtained from knowledge bases such as Wordnet~\citep{Fellbaum2000WordNetA}.
To incorporate contextual information, each sentence representation also contains the label of the previous sentence (``history feature'') and the sentence position in the document (``location feature'').
To better consider the interdependence between labels, the approaches usually apply CRFs. \citet{Asadi2019AutomaticZI} suggest fusion techniques instead to fuse the predicted label sequence within a dynamic window of sentences.
However, \citet{Asadi2019AutomaticZI}, \citet{Badie2018ZoneIB}, and \citet{Fisas2015OnTD} exploit the \textit{ground truth label} instead of the predicted label of the preceding sentence (``history feature'') during prediction (confirmed by the authors in e-mail correspondence), which has a significant impact on the performance.

Related tasks also classify sentences in full papers with deep learning methods,
e.g. for citation intent classification~\citep{Cohan2019StructuralSF,kunnath-etal-2020-overview},
or algorithmic metadata extraction~\citep{Safder2020} but without exploiting context from surrounding sentences.
Comparable to us, \citet{lauscher2018a} utilise a hierarchical deep learning architecture for argumentation mining in full papers but evaluate it only on one corpus.
To the best of our knowledge, a unified approach for sequential sentence classification for abstracts as well as full papers has not been proposed and evaluated yet.

\subsection{Transfer Learning}
The idea of transfer learning is to exploit knowledge from a source task in order to improve prediction accuracy in a target task.
The tasks can have training data from different domains and vary in their objectives.
According to Ruder's taxonomy for transfer learning ~\citep{Ruder2019Neural}, we investigate inductive transfer learning in this study since the target training datasets are labelled.
Inductive transfer learning can be further subdivided into multi-task learning, where tasks are learned simultaneously, and sequential transfer learning (also referred to as parameter initialisation), where tasks are learned sequentially.
Since there are so many applications for transfer learning, we focus on the most relevant cases for sentence classification in scientific texts.
For a more comprehensive overview on transfer learning in general and in NLP, we refer to \citet{Pan10} and \citet{Ruder2019Neural}.

\emph{Fine-tuning a pre-trained language model} such as BERT \citep{Devlin2018BERTPO}, Generative Pre-trained Transformer (GPT) \citep{Brown20}, or Universal Language Model Fine-tuning (ULMFiT) \citep{Ruder18} is a popular approach for sequential transfer learning in NLP.
Here, the source task consists of learning a language model (or a variant of it, e.g. predict masked words in a sentence) using a large unlabelled text corpus. Then, the parameters of the model are fine-tuned with labelled data of the target task.
\citet{edwards-etal-2020-go} evaluate the importance of domain-specific unlabelled data on pre-training word embeddings for text classification in the general domain (i.e. data such as news, phone conversations, magazines, etc.).
\citet{Pruksachatkun20} improve these language models by \emph{intermediate task transfer learning} where a language model is fine-tuned on a data-rich intermediate task before fine-tuning on the final target task.
\citet{park-caragea-2020-scientific} provide an empirical study on intermediate transfer learning from the general domain to scientific keyphrase identification. They show that SciBERT in combination with related tasks such as sequence tagging improves performance, while BERT or unrelated tasks degrade the performance.

For \emph{sequence tagging},
\citet{Yang2017TransferLF} investigate multi-task learning in the general domain with cross-domain, cross-application, and cross-lingual transfer. In particular, target tasks with few labelled data benefit from related tasks.
\citet{Lee2017TransferLF} successfully transfer pre-trained parameters from a big dataset to a small dataset in the biological domain.
\citet{SchulzEDKG18} evaluated multi-task learning for argumentation mining with multiple datasets in the general domain and could show that performance improves when training data for the tasks is sparse.
For \emph{coreference resolution}, \citet{Brack2021Coref} successfully apply sequential transfer learning and utilise a large dataset from the general domain to improve models for a small dataset in the scientific domain.

For \emph{sentence classification},
\citet{Mou2016HowTA} compare (1) transferring parameters from a source dataset to a target dataset against (2) training one model with two datasets in the general domain. They demonstrate that semantically related tasks improve while unrelated tasks degrade the performance of the target tasks.
\citet{Semwal2018} investigate the extent of task relatedness for product reviews and sentiment classification with sequential transfer learning.
\citet{Su20} evaluate multi-task learning for sentiment classification in product reviews from multiple domains.
\citet{LauscherGP018} evaluate multi-task learning on scientific texts, however, only on one dataset with different annotation layers.

Several approaches have been proposed to \emph{train multiple tasks jointly}:
\citet{Luan2018MultiTaskIO} train a model on three tasks (coreference resolution, entity and relation extraction) using one dataset of research papers.
\citet{Sanh2018AHM} introduce a multi-task model that is trained on four tasks (mention detection, coreference resolution, entity and relation extraction) with two different datasets.
\citet{Wei2019JointEO} utilise a multi-task model for entity recognition and relation extraction on one dataset in the general domain.
Comparable to us, \citet{Changpinyo18} analyse multi-task training with multiple datasets for sequence tagging.
In contrast, we investigate sequential sentence classification across multiple science domains.

\section{Cross-Domain Multi-task Learning for Sequential Sentence Classification}

On the one hand, the discussion of related work shows that several approaches and datasets from various scientific domains have been introduced for sequential sentence classification.
On the other hand, while transfer learning has been applied to various NLP tasks, it is known that the success depends largely on the relatedness of the tasks~\citep{Mou2016HowTA,Pan10,Ruder2019Neural}.
However, the transferability between datasets from different scientific domains has not been investigated yet and the field lacks a study on transfer learning for sequential sentence classification.
Furthermore, previous approaches investigated transfer learning for one or two datasets only.
To the best of our knowledge, a unified approach for different types of texts that differ noticeably by their structure and semantic context of sentences, as it is the case for abstracts and full papers, has not been proposed yet.

In this section, we propose a unified deep learning architecture for multi-task sequential sentence classification.
Our tailored transfer learning approaches, depicted in Figure~\ref{fig:proposed_approach}, exploit multiple datasets comprising different text types in form of abstracts and full papers. The unified approach without transfer learning is described in Section~\ref{sec:unified_approach} while Section~\ref{sec:transfer_methods} introduces the sequential transfer learning and multi-task learning approaches.
Finally in Section~\ref{sec:sem_relatedness}, we present an approach to semi-automatically identify the semantic relatedness of sentence classes between different annotation schemes.

\begin{figure*}[t]
    \center{\includegraphics[width=1.0\linewidth]
        {img/proposed_approach3.pdf}}
    \caption{Proposed approaches for sequential sentence classification: (a) unified deep learning architecture \emph{SciBERT-HSLN} for datasets including abstracts and full papers; (b) sequential transfer learning approaches, i.e. INIT 1 transfers all possible layers, INIT 2 only the sentence encoding layer; (c) and (d) are the multi-task learning approaches, i.e. in MULT ALL all possible layers are shared between the tasks, in MULT GRP the context enrichment is shared between tasks with the same text type.}
    \label{fig:proposed_approach}
\end{figure*}

\subsection{Unified Deep Learning Approach}
\label{sec:unified_approach}
Given a paper with the sentences ($s_1, ..., s_n$) and the set of dataset specific classes $L$ (e.g. \emph{Background}, \emph{Methods}, etc.), the task of \emph{sequential sentence classification} is to predict the corresponding label sequence ($y_1, ..., y_n$) with $y_i \in L$.
For this task, we propose a unified deep learning approach as depicted in Figure~\ref{fig:proposed_approach}(a), which is applicable to both abstracts \emph{and} full papers.
The core idea is to enrich sentence representations with context from surrounding sentences..

Our approach (denoted as \emph{SciBERT-HSLN}) is based on the \emph{Hierarchical Sequential Labeling Network (HSLN)} of \citet{Jin2018HierarchicalNN} (see Section~\ref{sec:approaches_abstract}). In contrast to \citet{Jin2018HierarchicalNN}, we utilise SciBERT~\citep{Beltagy2019SciBERTPC} as word embeddings and evaluate our approach on abstracts \emph{as well as} full papers.
SciBERT was pre-trained on scientific texts using the BERT architecture~\citep{Devlin2018BERTPO}.
We have chosen HSLN as the basis since it is better suited for full papers: it has no limitations on text length (in contrast to the approach of \citet{Cohan2019PretrainedLM}), and is computationally less expensive than the approach of \citet{yamada-etal-2020-sequential}.
Our \emph{SciBERT-HSLN} architecture has the following layers:

\paragraph{Word Embedding:} Input is a sequence of tokens $(t_{i,1}, ..., t_{i,m})$ of sentence $s_i$, while output is a sequence of contextual word embeddings $(w_{i,1}, ..., w_{i,m})$.

\paragraph{Sentence Encoding:}
Input  $(w_{i,1}, ..., w_{i,m})$ is transformed via a bidirectional long-short-term-memory (Bi-LSTM)~\citep{Hochreiter1997LongSM} into the hidden token representations $(h_{i,1}, ..., h_{i,m})$ ($h_{i,t} \in \mathbb{R}^{d^h}$) which are enriched with contextual information within the sentence.
Then, attention pooling~\citep{Jin2018HierarchicalNN,Yang2016HierarchicalAN} with $r$ heads produces a sentence vector $e_i \in \mathbb{R}^{rd^u}$.
An attention head produces a weighted average over the token representations of a sentence.
Multiple heads enable to capture several semantics of a sentence.
Formally, at first the token representation $h_{i,t}$ is transformed via a feed-forward network into a further hidden representation $a_{i,t}$ with the learned weight matrix $W^{[S]}$ and bias vector $b^{[S]}$:
\begin{eqnarray}
a_{i,t} = FFN(h_{i,t}) = \tanh (W^{[S]} h_{i,t} + b^{[S]})
\end{eqnarray}
Then, for each attention head $k$ with $1 \leq k \leq r$ the learned token level context vector $u_k \in \mathbb{R}^{d^u}$ is used to compute importance scores for all token representations which are then normalised by $softmax$:
\begin{eqnarray}
\alpha_{k,i,t} = \frac{\exp(u_k^{\intercal} a_{i,t})}{\sum_{t'} \exp(u_k^{\intercal} a_{i,t'})}
\end{eqnarray}
Afterwards, an attention head $e_{k,i} \in \mathbb{R}^{d^h}$ is computed as a weighted average over the token representations and all heads are concatenated to form the final sentence representation $e_i \in \mathbb{R}^{r d^h}$:
\begin{eqnarray}
e_{k,i} = \sum_{t'} \alpha_{k,i,t'} h_{i,t'} \\
e_{i} = \mbox{vec}([e_{1,i}, ..., e_{r,i}])
\end{eqnarray}

\paragraph{Context Enrichment:}
This layer takes as input all sentence representations $(e_1,...,e_n)$ of the paper and outputs contextualised sentence representations $(c_1,...,c_n)$ ($c_i \in \mathbb{R}^{d^h}$) via a Bi-LSTM. Thus, each sentence representation $c_i$ contains contextual information from surrounding sentences.

\paragraph{Output Layer:}
This layer transforms sentence representations $(c_1,...,c_n)$ via a linear transformation to the logits $(l_1, ..., l_n)$ with $l_i \in \mathbb{R}^{|L|}$.
Each component of vector $l_i$ contains a score for the corresponding label:
\begin{eqnarray}
l_i = W^{[O]} c_i + b^{[O]}
\end{eqnarray}
Finally, the logits serve as input for a conditional random field (CRF)~\citep{Lafferty2001ConditionalRF} that predicts the label sequence $(\hat y_1, ..., \hat y_n)$ ($\hat y_i \in L$) with the highest joint probability.
A CRF captures linear (one step) dependencies between the labels (e.g. \textit{Methods} are usually followed by \textit{Methods} or \textit{Results}). Therefore, a CRF learns a transition matrix $T \in \mathbb{R}^{|L| \times |L|}$, where $T_{l_1, l_2}$ represents the transition score from label $l_1$ to label $l_2$, and two vectors ${b, e \in \mathbb{R}^{|L|}}$, where $b_{l}$ and $e_{l}$ represent the score of beginning and ending with label $l$, respectively.
The objective is to find the label sequence with the highest conditional joint probability $P(\hat y_1, ..., \hat y_n|l_1, ..., l_n)$.
For this purpose, we define a score function for a label sequence $(\hat y_1, ..., \hat y_n)$, that is a sum of the scores of the labels and the transition scores:
\begin{eqnarray}
score((\hat y_1, ..., \hat y_n), (l_1, ..., l_n)) = b_{\hat y_1} + \sum_{t=1}^{n} l_{t,\hat y_t} + \sum_{t=1}^{n-1} T_{\hat y_t, \hat y_{t+1}} + e_{\hat y_m}
\end{eqnarray}
Then, the score is transformed to a probability value with $softmax$:
\begin{eqnarray}
Z(l_1, ..., l_n) = \sum_{y_1', ..., y_n'} exp(score((y_1', ..., y_n'), (l_1, ..., l_n))) \\
P(\hat y_1, ..., \hat y_n|l_1, ..., l_n) = \frac{exp(score((\hat y_1, ..., \hat y_n), (l_1, ..., l_n)))}{Z(l_1, ..., l_n)}
\end{eqnarray}
The denominator $Z(.)$ represents a sum of the scores of all possible label sequences for the given logits.
The Viterbi algorithm~\citep{Forney73viterbi} is used for efficient calculating of the sequence with the highest score and the denominator (both with time complexity $O(|L|^2 \cdot n)$).

During training, the CRF maximises $P(y_1, ..., y_n|l_1, ..., l_n)$ of the ground truth labels for all $m$ training samples $((x^{(1)}, y^{(1)}), ..., (x^{(m)}, y^{(m)}))$, where $x^{(i)}$ represents the sentences of paper $i$ and $y^{(i)}$ the corresponding ground truth label sequence.
Thus, the objective is to minimise the following loss function:
\begin{eqnarray}
L = -\frac{1}{m} \sum_{i=1}^{m} \log P(y^{(i)} | l^{(i)})
\end{eqnarray}

For regularisation, we use dropout after each layer.
Sci\-BERT is not fine-tuned, since it requires the training of 110 Mio. additional parameters.

\subsection{Transfer Methods}
\label{sec:transfer_methods}
For sequential sentence classification, we propose the following transfer methods.

\paragraph{Sequential Transfer Learning (INIT):}
The approach first trains the model for the source task and uses its tuned parameters to initialise the parameters for the target task.
Then, the parameters are fine-tuned with the labelled data of the target task. As depicted in Figure~\ref{fig:proposed_approach}(b), we propose two types of layer transfers.
\emph{INIT~1}: transfer parameters of \emph{context enrichment} and \emph{sentence encoding};
\emph{INIT~2}: transfer parameters of \emph{sentence encoding}. Other layers, except \emph{word embedding}, of the target task are initialised with random values.

\paragraph{Multi-Task Learning (MULT):}
Multi-task learning (MULT) aims for a better generalisation by simultaneously training samples in all tasks and sharing parameters of certain layers between the tasks.
As depicted in Figure~\ref{fig:proposed_approach}(c,d), we propose two multi-task learning architectures.
\emph{MULT ALL} shares all layers between the tasks except the \emph{output layers} so that the model learns a common feature extractor for all tasks.
However, full papers are much longer and have a different rhetorical structure than abstracts. Therefore, it is not beneficial to share the context enrichment layer between both dataset types. Thus, in \emph{MULT GRP} the \emph{context enrichment layers} are only shared between datasets with the same text type.
Formally, the objective is to minimise the following loss functions:
\begin{eqnarray}
 L_{\mbox{\tiny MULT ALL}} = \sum_{t \in T^A \cup T^F} L_t(\Theta^S, \Theta^C, \Theta^O_{t})  \\
\begin{aligned}
 L_{\mbox{\tiny MULT GRP}} = \sum_{t \in T^A} L_t(\Theta^S, \Theta^{C^A}, \Theta^O_{t})
+ \sum_{t \in T^F} L_t(\Theta^S, \Theta^{C^F}, \Theta^O_{t})
\end{aligned}
\end{eqnarray}
where $T^A$ and $T^F$ are the tasks for datasets containing abstracts and full papers;
$L_t$ is the loss function for task $t$;
the parameters
$\Theta^S$ are for sentence encoding,
$\Theta^C$, $\Theta^{C^A}$, $\Theta^{C^F}$ for context enrichment,
and $\Theta^{O}_{t}$ for the output layer of task $t$.

Furthermore, we propose the variants \emph{MULT ALL SHO} and \emph{MULT GRP SHO} that are applicable if all tasks share the same (domain-independent) set of classes.
\emph{MULT  ALL  SHO} shares all layers among all tasks. \emph{MULT  GRP SHO} shares the context enrichment and output layer only between tasks with the same text type. Formally, the objective functions are defined as:
\begin{eqnarray}
 L_{\mbox{\tiny MULT ALL SHO}} = \sum_{t \in T^A \cup T^F} L_t(\Theta^S, \Theta^C, \Theta^O)  \\
\begin{aligned}
 L_{\mbox{\tiny MULT GRP SHO}} = \sum_{t \in T^A} L_t(\Theta^S, \Theta^{C^A}, \Theta^{O^A})
+ \sum_{t \in T^F} L_t(\Theta^S, \Theta^{C^F}, \Theta^{O^F})
\end{aligned}
\end{eqnarray}

\subsection{Semantic Relatedness of Classes}
\label{sec:sem_relatedness}

Datasets for sentence classification have different domain-specific annotation schemes, that is different sets of pre-defined classes.
Intuitively, some classes have a similar meaning across domains, e.g. the classes \textit{Model} and \textit{Experiment} in the ART corpus are semantically related to \textit{Methods} in PubMed-20k (PMD) (see Table~\ref{table:datasets}).
An analysis of semantic relatedness can help consolidate different annotation schemes.

We propose machine learning models to support the identification of semantically related classes according to the following idea: if a model trained for PMD recognises sentences labelled with \textit{ART:Model} as \textit{PMD:Method}, and vice versa, then the classes \textit{ART:Model} and \textit{PMD:Method} can be assumed to be semantically related.

Let $T$ be the set of all tasks,
$L$ the set of all classes in all tasks,
$m_t(s)$ the label of sentence $s$ predicted by the model for task $t$, and
$S^l$ the set of sentences with the ground truth label $l$.
For each class $l \in L$ the corresponding semantic vector $v_l \in \mathbb{R}^{|L|}$ is computed as follows:
\begin{eqnarray}
v_{l,l'} = \frac{\sum_{t \in T, s \in S^l} {\mathbbm{1}(m_t(s) = l')}}{|S^l|}
\end{eqnarray}
where $v_{l, l'} \in \mathbb{R}$ is the component of the vector $v_l$ for class $l' \in L$ and
$\mathbbm{1}(p)$
is the indicator function that returns $1$ if $p$ is true and $0$ otherwise.
Intuitively, the semantic vectors concatenated vertically to a matrix represent a ``confusion matrix'' (see Figure~\ref{fig:pred_heatmap} as an example). Indeed, if we consider only one task, then the matrix is a confusion matrix.

Now, we define the semantic relatedness of two classes $k, l \in L$ using cosine similarity:
\begin{eqnarray}
\text{semantic\_relatedness}(k, l) = \cos(v_{k}, v_{l}) = \frac{v_{k}^\intercal \cdot v_{l}}{||v_{k}|| \cdot ||v_{l}||} \label{eq:semantic_relatedness}
\end{eqnarray}

\section{Experimental Setup}

This section describes the experimental evaluation of the proposed approaches, i.e. used datasets, implementation details, and evaluation methods.

\begin{table*}[t]
\centering
\caption{Characteristics of the benchmark datasets. The row "State of the art" depicts the best results for approaches that do not exploit the ground truth label of the preceding sentence during prediction: for PMD \protect\citep{yamada-etal-2020-sequential}, for NIC \protect\citep{Cohan2019PretrainedLM},
for DRI \protect\citep{Badie2018ZoneIB} (cf. Table 7), and for ART \protect\citep{liakata2012automatic}.}
\begin{tabular}{l|r|r|r|r}
             & \textbf{PMD}   & \textbf{NIC}       & \textbf{DRI} & \textbf{ART}       \\
\hline
Domains             & Biomedicine  & Biomedicine  & Computer &  Chemistry, \\
                   &              &              & Graphics &  Computational          \\
                   &              &              &          &  Linguistic          \\
\hline
Text Type &  Abstract   & Abstract & Full paper & Full paper           \\
\hline
\# Papers &  20.000   & 1.000 & 40 & 225           \\
\# Sentences &  235.892    & 9.771  & 8.777 & 34.680           \\
$\varnothing$ \# Sentences  & 12  & 10   & 219  &  154          \\
\# Classes &  5    & 6  & 5 & 11           \\
\hline
Classes & Background     & Background & Background & Background                        \\
        & Objective & Intervention & Challenge & Motivation                        \\
        & Methods & Study & Approach                                   & Hypothesis                        \\
        & Results & Population & Outcome                                    & Goal       \\
        & Conclusion & Outcome & FutureWork                                 & Object                       \\
        &                         & Other                                      &                                            & Experiment                        \\
        &                                      &                                            &                                            & Model                             \\
        &                                      &                                            &                                            & Method                            \\
        &                                      &                                            &                                            & Observation                            \\
        &                                      &                                            &                                            & Result                            \\
        &                                      &                                            &                                            & Conclusion                        \\

\hline
State of the art   & 93.1 & 84.7  & 72.5 & 51.6 \\
Original metric   & weighted F1 & weighted F1 & weighted F1 & accuracy \\
\end{tabular}
\label{table:datasets}
\end{table*}

\subsection{Investigated Datasets}
Table~\ref{table:datasets} summarises the characteristics of the investigated datasets, namely PubMed-20k (PMD)~\citep{Dernoncourt2017PubMed2R}, NICTA-PIBOSO (NIC) \citep{Kim2011AutomaticCO}, ART~\citep{Liakata2010CorporaFT}, and Dr. Inventor (DRI)~\citep{Fisas2015OnTD}.
The four datasets are publicly available and provide a good mix to investigate the transferability: They represent four different scientific domains; PMD and NIC cover abstracts and are from the same domain but have different annotation schemes; DRI and ART cover full papers but are from different domains and have different annotation schemes; NIC and DRI are rather small datasets, while PMD and ART are about 20 and 3 times larger, respectively; ART has a much finer annotation scheme compared to other datasets.
As denoted in Table~\ref{table:datasets}, the state-of-the-art results for ART are the lowest ones since ART has more fine-grained classes than the other datasets.
In contrast, best results are obtained for PMD: It is a large dataset sampled from PubMed, where authors are encouraged to structure their abstracts.
Therefore, abstracts in PMD are more uniformly structured than in other datasets, leading to better classification results.

\subsection{Implementation}
\label{sec:implementation}
Our approaches are implemented in PyTorch~\citep{Ketkar2017IntroductionTP}.
The Adaptive Moment Estimation (ADAM) optimiser~\citep{Kingma2014AdamAM} with $0.01$ weight decay and an exponential learning rate decay of $0.9$ after each epoch is used for training.
To speed up training, sentences longer than 128 tokens are truncated since the computational cost for the attention layers in BERT is quadratic in sentence length~\citep{VaswaniSPUJGKP17}.
To reproduce the results of the original HSLN architecture, we tuned SciBERT-HSLN for PMD and NIC with hyperparameters as proposed in other studies ~\citep{Devlin2018BERTPO,Jin2018HierarchicalNN}.
The following parameters performed best on the validation sets of PMD and NIC: learning rate  3e-5, dropout rate $0.5$, Bi-LSTM hidden size $d^h=2 \cdot 758$, $r=15$ attention heads of size $d^u=200$.
We used these hyperparameters in all of our experiments.

For each dataset, we grouped papers to mini-batches without splitting them, if the mini-batch does not exceed 32 sentences. Thus, for full papers a mini-batch may consist of sentences from only one paper.
During multi-task training we switched between the mini-batches of the tasks by applying proportional sampling~\citep{Sanh2018AHM}.
After a mini-batch, only task-related parameters are updated, i.e. the associated output layer and all the layers below.

\subsection{Evaluation}
To be consistent with previous results and due to non-determinism in deep neural networks~\citep{Reimers2017ReportingSD}, we repeated the experiments and averaged the results.
According to \citet{Cohan2019PretrainedLM} we performed three random restarts for PMD and NIC and used the same train/validation/test sets.
For DRI and ART, we performed 10-fold and 9-fold cross-validation, respectively, as in the original papers \citep{Fisas2015OnTD,liakata2012automatic}.
Within each fold the data is split into train/validation/test sets with the proportions $\frac{k-2}{k}$/$\frac{1}{k}$/$\frac{1}{k}$ where $k$ is the number of folds.
For multi-task learning, the experiment was repeated with the maximum number of folds of the datasets used, but at least three times.
All models were trained for 20 epochs. The test set performance within a fold and restart, respectively, was calculated for the epoch with the best validation performance.

We compare our results only with approaches which do not exploit \emph{ground truth labels} of the preceding sentence as a feature \emph{during prediction} (see Section~\ref{sec:ml_approaches}).
This has a significant impact on the performance: Using the ground truth label of the previous sentences as a sole input feature to a SVM classifier already yields an accuracy of 77.7 for DRI and 55.5 for ART~\footnote{Compare also results for the ``history'' feature in~\citet{Badie2018ZoneIB} (cf. Table 5).}.
Best reported results using ground truth labels as input features have an accuracy of 84.15 for DRI and 65.75 for ART~\citep{Asadi2019AutomaticZI}.
In contrast, we pursue a realistic setting by exploiting the \emph{predicted} (not ground truth) label of neighbouring sentences during prediction.

Furthermore, we provide results for three strong deep learning baselines\footnote{We do not provide baseline results for the approach of \citet{yamada-etal-2020-sequential} since their implementation is not publicly available.} (see Section~\ref{sec:approaches_abstract}):
(1) fine-tuning SciBERT using the [CLS] token of individual sentences as by~\citet{Devlin2018BERTPO} (referred to as SciBERT-[CLS]),
(2) original HSLN implementation of \citet{Jin2018HierarchicalNN}, and
(3) the SciBERT-based approach of \citet{Cohan2019PretrainedLM}.

\begin{table}[t!]
\centering
\caption{Experimental results for the proposed approaches: our SciBERT-HSLN model without transfer learning, parameter initialisation (INIT), and multi-task learning (MULT ALL and MULT GRP). Previous state of the art (see Table~\ref{table:datasets}), SciBERT-[CLS], original HSLN approach of \citet{Jin2018HierarchicalNN}, and the approach of Cohan et. al.~\citep{Cohan2019PretrainedLM} are the baseline results. For PMD (P), NIC (N), and DRI (D) we report weighted F1 score and for ART (A) accuracy. $\varnothing$ denotes the average of all scores. \textit{Italics} depicts whether the result is better than the baseline, \textbf{bold} whether the transfer method improves SciBERT-HSLN, \underline{underline} the best overall result.}
\begin{tabular}{l|r|r|r|r|r}
                                           & \textbf{   PMD    } & \textbf{    NIC    } & \textbf{    DRI    }   & \textbf{    ART    }   & $\varnothing$ \\
                                           \hline
\textbf{Previous state of the art}                                       & \underline{93.1}       & 84.8  & 72.5  & 51.6 & 75.5 \\
SciBERT-[CLS]                                       & 89.6       & 78.4  & 69.5  & 51.5 & 72.3 \\
\citet{Jin2018HierarchicalNN} (HSLN)                                & 92.6       & 84.7  & 75.3  & 49.3  & 75.5  \\
\citet{Cohan2019PretrainedLM}           & 92.9       & 84.8  & 74.3  & 54.3 & 76.6 \\
\hline
\hline
\textbf{SciBERT-HSLN}                                & 92.9      & \textit{84.9} & \textit{78.0} & \textit{58.0} & \textit{78.5} \\
\hline
\hline
INIT 1 PMD to $T$                           & -          & 84.8 & \textbf{\textit{81.2}} & \textit{57.7} \\
INIT 2 PMD to $T$                           & -          & 84.8 & \textbf{\textit{80.1}} & \textit{58.0} \\
\hline
INIT 1 NIC to $T$                           &           92.9  &- & \textbf{\textit{81.9}}  &  \textit{57.6} \\
INIT 2 NIC to $T$                           &           92.9  &- & \textbf{\textit{79.6}}  &  \textit{57.2} \\
\hline
INIT 1 DRI to $T$                           &           92.9 & 83.5 & -  &  \textit{57.8} \\
INIT 2 DRI to $T$                           &           92.9 & 83.8 & -  &  \textit{57.6} \\
\hline
INIT 1 ART to $T$                           &           \textbf{93.0} & 84.7 & \textbf{\textit{82.2}}  &  - \\
INIT 2 ART to $T$                           &           92.9 & 84.7 & \textbf{\textit{81.0}}  &  - \\
\hline
\hline
\textbf{MULT ALL}                                   & \textbf{93.0}      & \textbf{\textit{86.0}} & \textbf{\textit{81.8}} & \textit{57.7} & \textbf{\textit{79.6}}\\
\hline

PMD, NIC                         & \textbf{93.0}      & \textit{\textbf{86.1}} & -     & -     \\
PMD, DRI                           & 92.9      & -     & \textbf{\textit{80.6}}  & -     \\
PMD, ART                           & \textbf{93.0}      & -     & -     & \textit{58.0} \\

NIC, DRI                          & -          & 84.2  & \textbf{\textit{80.7}} & -     \\
NIC, ART                          & -          & 84.4 & -     & \textit{57.9} \\
DRI, ART                            & -          & -     & \textbf{\textit{82.0}}    & \textit{57.6} \\

\hline

PMD, NIC, DRI                    & \textbf{93.0}      & \textit{\textbf{86.2}} & \textbf{\textit{81.0}} & -     \\
PMD, NIC, ART                     & \textbf{93.0}          & \underline{\textbf{\textit{86.3}}}  & - & \textit{58.0} \\
PMD, DRI, ART                     & \textbf{93.0}          &   -& \textbf{\textit{82.7}} & \textit{57.8} \\
NIC, DRI, ART                     & -          & 84.7 & \textbf{\textit{82.0}} & \textit{57.7} \\

\hline
\hline
\textbf{MULT GRP}  & \textbf{93.0}  & \textit{\textbf{86.1}} & \textbf{\textit{83.4}} & \underline{\textbf{\textit{58.8}}} & \underline{\textbf{\textit{80.3}}} \\
\hline
 P,N,D,A  & 92.9  & \textit{\textbf{85.4}} &  \underline{\textbf{\textit{84.4}}} & \textit{58.0} & \textbf{\textit{80.2}}\\
\hline
(P,D),(N,A)  & \textbf{93.0}  & \textit{\textbf{86.0}} & \textbf{\textit{81.1}} & \textbf{\textit{58.5}} & \textbf{\textit{79.7}} \\
(P,A),(N,D)  & 92.9  & \textit{\textbf{85.8}} & \textbf{\textit{83.6}} & \textit{58.0} & \textbf{\textit{80.1}} \\
\hline

 (P,N,D),(A)  & 92.9  & \textit{\textbf{86.0}} & \textbf{\textit{80.6}} & \textbf{\textit{58.2}} & \textbf{\textit{79.4}} \\
 (P,N,A),(D)  & \textbf{93.0}  & \textit{\textbf{86.0}} & \textbf{\textit{84.1}} & \textbf{\textit{58.1}} & \textbf{\textit{80.3}} \\
(P,D,A),(N)  & 92.9  & \textit{\textbf{85.5}} & \textbf{\textit{82.2}} & \textit{58.0} & \textbf{\textit{79.6}} \\
 (N,D,A),(P)  & 92.9  & \textit{\textbf{85.9}} & \textbf{\textit{83.3}} & \textbf{\textit{58.5}} & \textbf{\textit{80.1}} \\

\end{tabular}
\label{table:results}
\end{table}

\section{Results and Discussion}
In this section, we present and discuss the experimental results for our proposed cross-domain multi-task learning approach for sequential sentence classification.
The results for different variations of our approach as well as the baseline are depicted in Table~\ref{table:results}.
The results are discussed in the following three subsections with regard to the unified approach without transfer learning (Section \ref{sec:notransfer}), with transfer learning (Section \ref{sec:transferinit}), and multi-task transfer learning (Section \ref{sec:transfermulti}).
Section~\ref{sec:analysis_semantic_relatedness} analyses the semantic relatedness of classes for the four annotation schemes.

\subsection{Unified Approach without Transfer Learning (SciBERT-HSLN)}
\label{sec:notransfer}
For the PMD and NIC datasets, we can achieve the current state-of-the-art results reported by \citet{yamada-etal-2020-sequential} and \citet{Cohan2019PretrainedLM} with our SciBERT-HSLN model.
Thus, our proposed approach is competitive with the current approaches for sequential sentence classification in abstracts.

For the full paper datasets DRI and ART, our SciBERT-HSLN model significantly outperforms the previously reported best results and the deep learning baselines SciBERT-[CLS], HSLN, and the approach of \citet{Cohan2019PretrainedLM}.
The previous state of the art for DRI and ART~\citep{Badie2018ZoneIB,liakata2012automatic} requires feature engineering, and a sentence is enriched only with context of the previous sentence. In SciBERT-[CLS] each sentence is classified in isolation.
The original HSLN architecture \citep{Jin2018HierarchicalNN} uses shallow word embeddings pre-trained on biomedical texts. Thus, incorporating SciBERT contextual word embeddings into HSLN helps improve performance for the DRI and ART datasets.
The approach of \citet{Cohan2019PretrainedLM} can process only about 10 sentences at once since SciBERT supports sequences of up to 512 tokens only. Thus, long text has to be split into multiple chunks. Our deep learning approach can process \emph{all} sentences of a paper at once so that all sentences are enriched with context from surrounding sentences.

\emph{Our unified deep learning approach is applicable to datasets consisting of different text types, i.e. abstracts and full papers, without any feature engineering (RQ7).}

\subsection{Sequential Transfer Learning (INIT)}
\label{sec:transferinit}
Using the INIT approach, we can only improve the baseline results for the DRI dataset in all settings.
The approach INIT~1 performs better than INIT~2 in most cases which indicates that transferring all parameters is more effective.

\emph{However, the results suggest that sequential transfer learning is not a very effective transfer method for sequential sentence classification (RQ2).}

\subsection{Multi-Task Learning (MULT)}
\label{sec:transfermulti}
Next, we discuss the results of our multi-task learning approach, and the effects of multi-task learning on smaller datasets and individual sentence classes.

\paragraph{MULT ALL:}
All tasks were trained jointly sharing all possible layers.
Except for the ART task, all results are improved using the SciBERT-HSLN model.
For the PMD task the improvement is marginal, since the baseline results (F1 score) were already on a high level.
Pairwise MULT ALL combinations show that the models for PMD and NIC, respectively, benefit from the (respective) other dataset, and the DRI model especially from the ART dataset.
The PMD and NIC datasets are from the same domain, and both contain abstracts, so the results are as expected. Furthermore, DRI and ART datasets both contain full papers, and DRI has more coarse-grained classes. However, ART is a related large dataset with fine-grained classes and presumably therefore the model for ART does not benefit from other datasets.
In triple-wise MULT ALL combinations the models for PMD and DRI, respectively, benefit from all datasets, and the model for NIC only if the PMD dataset is present.

\emph{The results suggest that sharing all possible layers between multiple tasks is effective except for bigger datasets with more fine-grained classes (RQ3, RQ4).}

\paragraph{MULT GRP:}
In this setting, the models for all tasks were trained jointly, but only models for the same text type share the \emph{context enrichment layer}, i.e. (PMD, NIC) and (DRI, ART).
Here, all models benefit from the other datasets.
In our ablation study, we also provide results for sharing only the \emph{sentence encoding layer}, referred as MULT GRP P,N,D,A, and all pairwise and triple-wise combinations sharing the \emph{context enrichment layer}.
Other combinations also yield good results. However, MULT GRP is effective for \emph{all} tasks.
\emph{Our results indicate that sharing the sentence encoding layer between multiple models is beneficial. Furthermore, sharing the context enrichment layer only between models for the same text type is an even more effective strategy (RQ3, RQ4).}

\paragraph{Effect of Dataset Size:}
The NIC and DRI models benefit more from multi-task learning than PMD and ART.
However, PMD and ART are bigger datasets than NIC and DRI.
The ART dataset has also more fine-grained classes than the other datasets. This raises the following question:

\emph{How would the models for PMD and ART benefit from multi-task learning if they were trained on smaller datasets?}

To answer this question, we created smaller variants of PMD and ART, referred to as $\mu$PMD and $\mu$ART, with a comparable size with NIC and DRI.
Within each fold we truncated the training data to $\frac{1}{20}$ for $\mu$PMD and $\frac{1}{3}$ for $\mu$ART while keeping the original size of the validation and test sets.
As shown in Table~\ref{table:small_results} all models benefit from the other datasets, whereas MULT GRP again performs best.
\emph{The results indicate that models for small datasets benefit from multi-task learning independent of the difference in the granularity of the classes (RQ1).}

\paragraph{Effect for each Class:}
Figure~\ref{fig:f1_per_label} shows the F1 scores per class for the investigated approaches.
Classes, which are intuitively highly semantically related (*:Background, *:Results, *:Outcome), and classes with few examples (DRI:FutureWork, DRI:Challenge, ART:Hypothesis, NIC:Study Design) tend to benefit significantly from multi-task learning.
The classes ART:Model, ART:\-Observation, and ART:Result have worse results than SciBERT-HSLN when using MULT ALL, but MULT GRP yields better results.
This can be attributed to sharing the \emph{context enrichment layers} only between datasets with the same text type.
\emph{The analysis suggests that especially semantically related classes and classes with few examples benefit from multi-task learning (RQ1).}

\begin{table}[t]
\centering
\caption{Experimental results for $\mu$PMD, NIC, DRI and $\mu$ART with our SciBERT-HSLN model and our proposed multi-task learning approaches.}
\begin{tabular}{l|r|r|r|r|r}
         & \textbf{$\mu$PMD} & \textbf{NIC} & \textbf{DRI}   & \textbf{$\mu$ART} & $\varnothing$  \\
\hline
SciBERT-HSLN & 90.9  & 84.9 & 78.0 & 52.2 & 76.5\\
MULT ALL & \textbf{91.1}  & \textbf{85.7} & \textbf{81.0} & \textbf{53.8} & \textbf{77.9} \\
MULT GRP   & \textbf{91.1}  & \textbf{85.9} & \textbf{82.2} & \textbf{55.1} & \textbf{78.6}
\end{tabular}
\label{table:small_results}
\end{table}

\begin{figure}[t]
    \centering
    \subfloat{\includegraphics[width=0.50\linewidth]
        {img/f1_per_label_1_ord.pdf}}
    \hspace{-2.3em}
    \qquad
    \subfloat{\includegraphics[width=0.50\linewidth]
        {img/f1_per_label_2_ord.pdf}}
    \caption{F1 scores per class for the datasets PMD, NIC, DRI, and ART for the approaches SciBERT-HSLN, MULT ALL, MULT GRP, and the best combination for the respective dataset. Numbers at the bars depict the F1 scores of the best classifiers and in brackets the number of examples for the given class. The classes are ordered by the number of examples.}
    \label{fig:f1_per_label}
\vspace{-1em}
\end{figure}

\begin{figure}[t]
    \center{\includegraphics[width=1.0\linewidth]
        {img/multi_heatmap2.pdf}}
    \caption{Each row represents a semantic vector representation as described in Section~\ref{sec:sem_relatedness} for a class computed with \emph{MULT ALL} classifier.}
    \label{fig:pred_heatmap}
\end{figure}

\subsection{Semantic Relatedness of Classes across Annotation Schemes}
\label{sec:analysis_semantic_relatedness}
In this section, we first evaluate our proposed approach to semi-automatically identify semantically related classes in the investigated datasets PMD, NIC, DRI, and ART. Based on our analysis, we identify six clusters of semantically related classes.
Then, we present a new dataset that is compiled from the investigated datasets and is based on the identified clusters.
As a possible down-stream application, this multi-domain dataset with a generic set of classes could help to structure research papers in a domain-independent manner, supporting, for instance, the development of academic search engines.

\begin{figure}[t]
    \centering
    \subfloat[SciBERT-HSLN classifiers without transfer learning]{\includegraphics[width=0.48\linewidth]
        {img/labels_correlations_base.pdf}}
    \qquad
    \hspace{-2em}
    \subfloat[\emph{MULT GRP} classifier]{\includegraphics[width=0.48\linewidth]
        {img/labels_correlations_groups.pdf}}
    \qquad{}
    \subfloat[\emph{MULT ALL} classifier]{\includegraphics[width=0.80\linewidth]
        {img/labels_correlations_multi.pdf}}
    \caption{Semantic vectors of classes computed by (a) SciBERT-HSLN without transfer learning, (b) \emph{MULT GRP} and (c) \emph{MULT ALL}, and projected to 2D space via PCA.}
    \hspace{-3em}
    \label{fig:label_correlations}
\end{figure}

\paragraph{Analysis of Semantic Relatedness of Classes:}
Based on the annotation guidelines of the investigated datasets PMD~\citep{Dernoncourt2017PubMed2R}, NIC~\citep{Kim2011AutomaticCO}, DRI~\citep{Fisas2015OnTD}, and ART~\citep{Liakata2010CorporaFT}, we identified six clusters of semantically related classes, which are depicted in Figure~\ref{fig:label_correlations}.
The identification process of the clusters followed the intuition, that most research papers independent of the scientific domain (1) investigate a research problem (\emph{Problem}), (2) provide background information for the problem (\emph{Background}), (3) apply or propose certain methods (\emph{Methods}), (4) yield results (\emph{Results}), (5) conclude the work (\emph{Conclusions}), and (6) outline future work (\emph{Future Work}).

Figure~\ref{fig:pred_heatmap} shows the semantic vectors for all classes computed with the MULT ALL model. It can be observed that some semantic vectors look similar, e.g. PMD:Background and DRI:Background.
We computed the semantic vectors also with SciBERT-HSLN and MULT GRP and projected them onto a 2D space via principal component analysis~\citep{Jolliffe2011PrincipalCA}, as shown in Figure~\ref{fig:label_correlations}.
It can be seen that already for the SciBERT-HSLN classifiers our approach enables to identify semantically related classes (e.g. see \emph{Results} cluster).
However, the MULT ALL model yields more meaningful clusters.
Except \emph{Problem}, all clusters for semantically related classes are well identifiable in Figure~\ref{fig:label_correlations}(c).
Although MULT GRP performs best, the clusters are not consistent in Figure~\ref{fig:label_correlations}(b).
The semantic vector for ART:Hypothesis is an outlier in the \emph{Problem} cluster in Figure~\ref{fig:label_correlations}(c), since ART:Hypothesis is confused mostly with ART:Conclusion and ART:Result (see Figure~\ref{fig:pred_heatmap}) and has also a very low F1 score (see Figure~\ref{fig:f1_per_label}).

\begin{table}[t]
\center
\caption{Silhouette scores per cluster and overall computed for the semantic vectors of SciBERT-HSLN, MULT GRP and MULT ALL classifiers.}
\label{tab:clusters_Silhouette}
\begin{tabular}{l|r|r|r}
                & SciBERT & MULT  & MULT \\
                &   HSLN &  GRP & ALL \\ \hline
Background      & 0.45         & 0.18     & \textbf{0.48}     \\
Problem         & -0.27        & \textbf{-0.04}    & -0.29    \\
Methods         & 0.19         & -0.03    & \textbf{0.31}     \\
Results         & -0.38        & 0.01     & \textbf{0.32}     \\
Conclusions     & \textbf{0.92}         & -0.49    & 0.02     \\
Future Work     & 0.00         & 0.00     & 0.00     \\
\hline
Overall  & 0.10         & -0.02    & \textbf{0.20}
\end{tabular}
\end{table}

Table~\ref{tab:clusters_Silhouette} shows the Silhouette scores~\citep{Rousseeuw1987} for each cluster.
A positive Silhouette score indicates that objects lie well within the cluster, and a negative score that the objects are merely somewhere in between clusters.
As a distance metric, we use $\text{semantic\_relatedness}$ as defined in Equation~\ref{eq:semantic_relatedness}.
The Silhouette scores also confirm that MULT ALL forms better clusters than SciBERT-HSLN and MULT GRP.
We hypothesise that MULT ALL can better capture the semantic relatedness of classes than the other approaches since it is enforced to learn a generic feature extractor across multiple datasets.

\emph{The multi-task learning approach sharing all possible layers is able to recognise semantically related classes (RQ5).}

\paragraph{Domain-Independent Sentence Classification:}
Based on the identified clusters, we compile a new dataset \emph{G-PNDA} from the investigated datasets PMD, NIC, DRI, and ART.
The labels of the datasets are collapsed according to the clusters in Figure~\ref{fig:label_correlations}.
Table~\ref{table:generic_datasets} summarises the characteristics of the compiled dataset.
To prevent a bias towards bigger datasets, we truncate PMD to $\frac{1}{20}$ and ART to $\frac{1}{3}$ of their original size.

Table~\ref{tab:generic_results} depicts our experimental settings and results for the generic dataset \emph{G-PNDA}.
We train a model for each dataset part, and the multi-task learning models MULT ALL and MULT GRP.
Since we have common sentence classes now, we train also models that share the output layers between the dataset parts, referred to as MULT ALL SHO and MULT GRP SHO (see Section~\ref{sec:transfer_methods}).
For training and evaluation, we split each dataset part into train/validation/test sets with the portions 70/10/20, average the results over three random restarts and use the same hyperparameters as before (see Section~\ref{sec:implementation}).

Table~\ref{tab:generic_results} shows that the proposed MULT GRP model outperforms all other settings.
Surprisingly, sharing the output layer impairs the performance in all settings.
We can attribute this to the fact that the output layer learns different transition distributions between the classes.

\emph{Thus, in a domain-independent setting a separate output layer per dataset part helps the model to capture the individual rhetorical structure present in the domains (RQ3, RQ6).}

\begin{table}[tb]
\centering
\caption{Characteristics of the domain-independent dataset \emph{G-PNDA} that was compiled from the origin datasets \emph{PMD}, \emph{NIC}, \emph{DRI}, and \emph{ART}.}
\begin{tabular}{l|r|r|r|r}
             & \textbf{G-PMD}   & \textbf{G-NIC}       & \textbf{G-DRI} & \textbf{G-ART}       \\
\hline
Text Type &  Abstract   & Abstract & Full paper & Full paper           \\
\hline
\# Papers &  1.000   & 1.000 & 40 & 67           \\
\# Sentences &  11.738    & 9.771  & 8.777 & 9.528           \\
$\varnothing$ \# Sentences  & 11  & 10   & 219  &  142          \\
\hline
Background          & 1.220 & 2.548 & 1.760     & 1.657  \\
Problem             & 953   & 0     & 449   & 529  \\
Methods             & 3.927 & 2.700 &  5.038     & 2.752  \\
Results              & 3.760 & 4.523 & 1.394 & 3.672  \\
Conclusions          & 1.878 & 0     & 0     & 918  \\
Future Work         & 0     & 0     & 136   & 0  \\
\hline
\end{tabular}
\label{table:generic_datasets}
\end{table}

\begin{table}[tb]
\caption{Experimental results in terms of F1 scores for our proposed approaches for the generic dataset \emph{G-PNDA}:
baseline model SciBERT-HSLN with one separate model per dataset and the multi-task learning models MULT ALL SHO, MULT ALL, MULT GRP SHO, and MULT GRP.
\textbf{Bold} depicts whether the approach improves the baseline, \underline{underline} the best overall result.}
\label{tab:generic_results}
\begin{tabular}{l|r|r|r|r|r}
                                 & G-PMD & G-NIC & G-DRI & G-ART & $\varnothing$ \\
\hline
SciBERT-HSLN                         & 90.1 & 89.3 & 81.7 & 70.8 & 83.0 \\
(one model per dataset)          &      &       &        &       & \\
\hline \hline
MULT ALL SHO                         & 89.8 & 89.1 & \textbf{83.5} & 67.1 & 82.4 \\
(shared output layer)               &      &       &        &     & \\
\hline
MULT ALL                       & \textbf{90.5} & \textbf{89.8}  & \textbf{84.9} & 70.5 & \textbf{83.9} \\
(separate output layer)         &      &       &        &  & \\
\hline \hline
MULT GRP SHO                        & 90.0 & \underline{\textbf{89.9}} & \textbf{86.1} & 70.4 & \textbf{84.1} \\
(shared output layer)          &      &       &        &   & \\
\hline
MULT GRP                        & \underline{\textbf{90.6}} & \textbf{89.7} & \underline{\textbf{87.2}} & \underline{\textbf{71.0}} & \underline{\textbf{84.6}} \\
(separate output layer)         &      &       &        &  &
\end{tabular}
\end{table}

\section{Conclusions}

In this paper, we have presented a unified deep learning architecture for the task of sequential sentence classification.
The unified approach can be applied to different types of text with a differing structutre, e.g. abstracts as well as full papers.
For datasets consisting of full papers, our approach significantly outperforms the state of the art without any feature engineering, while being competitive for datasets consisting of abstracts only.

Furthermore, we have tailored two common transfer learning approaches to sequential sentence classification and compared their performance.
We found that training a multi-task model with multiple datasets works better than sequential transfer learning.
Our comprehensive experimental evaluation with four different datasets offers useful insights under which conditions transferring or sharing of specific layers is beneficial or not.
In particular, it is always beneficial to share the sentence encoding layer between datasets from different domains.
However, it is most effective to share the context enrichment layer, which encodes the context of neighbouring sentences, only between datasets with the same text type (abstracts vs. full papers).
This can be attributed to different rhetorical structures in abstracts and full papers.
Our tailored multi-task learning approach makes use of multiple datasets and yields new state-of-the-art results for three of the four examined datasets.
In particular, models for tasks with small datasets and classes with few labelled examples benefit significantly from models of other tasks.

Our study suggests that the classes of the different dataset annotation schemes are semantically related, even though the datasets come from different domains and have different text types. This semantic relatedness is an important prerequisite for transfer learning in NLP tasks~\citep{Mou2016HowTA,Pan10,Ruder2019Neural},

Finally, we have proposed an approach to semi-automatically identify semantically related classes from different datasets to support manual comparison and inspection of different annotation schemes across domains. We demonstrated the usefulness of the approach with an analysis of four annotation schemes. This approach can support the investigation of annotation schemes across disciplines without re-annotating datasets.
From the analysis, we have derived a domain-independent consolidated annotation scheme and compiled a domain-independent dataset. This allows for the classification of sentences in research papers with generic classes across disciplines, which can support, for instance, academic search engines.

In future work, we plan to incorporate general, and domain-specific knowledge bases into the architecture and integrate further tasks into the multi-task learning approach.
Furthermore, we intend to evaluate the domain-independent sentence classifier in an information retrieval scenario.

\section*{Conflict of interest}
The authors declare that they have no conflict of interest.

\bibliographystyle{spbasic}


\section{Introduction}
Entanglement is a striking feature of quantum systems and
responsible for many quantum tasks such as teleportation, dense
coding, key distribution, error correction etc. \cite{nielsen},
which has provided a strong motivation for the study of detection
and quantification of entanglement. There have been many results
related to separability criterion and entanglement measures whose
effectiveness depends on detailed quantum states. For instance PPT
criterion \cite{Peres A.,M. Horodecki} detects many entangled states
but not bound entangled ones, while the realignment criterion does
\cite{K.Chen}. The entanglement of formation \cite{eof,M.B.P} and
concurrence \cite{con,P.Rungta,S.A} are two well defined
quantitative measures of quantum entanglement. However the
entanglement of formation and concurrence have only explicit
analytical results for some special quantum states such as Werner
states and isotropic states \cite{Terhal-Voll2000,S.M1,S.M2,S.M3,P.
Rungta11}.

For multipartite case there are two well known classes of pure
states, the GHZ and W states. They are shown to be robust against
external flux fluctuations for feasible experimental realizations
\cite{Mun Dae Kim} and the related fidelity can be determined with
an effort increasing only linearly with the number of qubits
\cite{O. Guhne}. Two-party and three-party quantum teleportation
with GHZ state has been discussed. The W state can be also used as
quantum channels for perfect two-party teleportation \cite{A.
Karlsson,W. K. Wootters,P. Agrawal} and quantum key distribution
\cite{J. Joo}. In \cite{Afshin Montakhab} the entanglement dynamics
of GHZ state and W state have been monitored under different models
of system-environment interaction and an exponential decay of
entanglement as a function of time has been obtained. In \cite{B.
Fortescue} a protocol has been presented for distilling maximally
entangled bipartite states between random pairs of parties from
those sharing a tripartite W state. Various experiments have been
set up in the literature for generating three-qubit GHZ and W states
by applying optical systems, nuclear magnetic resonance, cavity QED,
or ion trapping techniques. In \cite{H.H}, they report the scalable
and deterministic generation of four-, five-, six-, seven- and
eight- particle entangled states of $W$ state with trapped ions.

The Schmidt-correlated (SC) states are the mixtures of pure states,
sharing the same Schmidt bases \cite{Rains}. They are generalized to
multipartite case, having the generalized GHZ states as pure state
decompositions in \cite{ming}. An $N$-partite SC state
$\rho_{SC}\in\bb{C}^M\otimes \cdots \otimes \bb{C}^M$ is generally
of the form
\begin{equation}\label{sc}
\rho_{SC} = \sum_{m,n=0 }^{M-1} \rho_{mn} |m \cdots m \rangle \langle n
\cdots n|,
\end{equation}
where $\sum_{m=0}^{M-1} \rho_{mm}=1$. For any pure state
decomposition $\rho_{SC}=\sum_k p_k |\phi_k \rangle\langle \phi_k|$,
$|\phi_k \rangle$ has the from $|\phi_k \rangle =\sum_m \sqrt
{\rho_{mm}} e^{ i \Theta_m^{(k)}} |m \cdots m \rangle $, which is a
kind of generalized $N$-partite $\rm GHZ(N,M)$ state, where
\begin{equation}\label{nmghz}
{\rm GHZ}(N,M) = \frac{1}{\sqrt{M}} (|0 \cdots 0 \rangle + |1 \cdots
1\rangle + \cdots + |M-1, \cdots, M-1 \rangle ).
\end{equation}
An SC state is fully separable if and only if it is PPT with respect
to some subsystems \cite{ming}, and it is either fully separable or
genuine entangled. Here we say an $N$-partite state $\rho$ is fully
separable if it can be written as
\begin{equation}\label{deffullysep}
 \rho=\sum_j p_j \rho^{(j)}_1 \otimes \rho^{(j)}_2 \otimes \cdots \otimes
 \rho^{(j)}_N.
\end{equation}
If $\rho$ is not fully separable, it may be biseparable, which means
it can be written as
\begin{equation}\label{def2sep}
 \rho=\sum_{perm\{i_1, i_2,
\cdots, i_N\}, m, j } p^{(j)}_{i_1, \cdots, i_N,m} \rho^{(j)}_{i_1,
i_2, \cdots, i_m} \otimes \rho^{(j)}_{i_{m+1}, \cdots, i_N},
\end{equation}
where $perm\{i_1, i_2, \cdots, i_N\}$ is a sum over all possible
permutations of the set of indices and $\sum_{perm\{i_1, i_2,
\cdots, i_N\}, m, j} p^{(j)}_{i_1, \cdots, i_N,m}=1$. Here
$\rho^{(j)}_{i_1, i_2, \cdots, i_m}$, $\rho^{(j)}_{i_{m+1}, \cdots,
i_N}$ are density
 matrices associated with the subsystems ${i_1, i_2, \cdots, i_m}$ and
${i_{m+1}, \cdots, i_N}$. We say that $\rho$ is genuine entangled if
it can not be written in the form of (\ref{def2sep}) \cite{O.G}.

In this paper we study
another class of multipartite mixed states, which have the
generalized W states as pure state decompositions.

\section{Generalized mixed W states in multiqubits system }

First we consider multipartite qubit case.
The $N$-partite $|W_N\ra$ state reads,
\begin{equation}\label{wn}
|W_N\ra=\frac{1}{\sqrt{N}}(|0 \cdots 01 \ra + |0 \cdots 10\ra +
\cdots + |1\cdots 00 \ra).
\end{equation}
The $|W_N\ra$ state corresponds to the Dicke state $|D_{1,N}\ra$ and
is therefore an example of a Dicke state. Dicke states have been
investigated already in 1954 by R. H. Dicke, while studying light
emission from a cloud of atoms. In fact they are simultaneous
eigenstates of the collective angular momentum operators
$J_z=\frac{1}{2}\sum_k \sigma_z^{(k)}$ and $J^2$ \cite{O.G}.

The generalized $|W_N\ra$ state is given by,
\begin{equation}\label{gw}
|W_N\ra_g=\sum_{m=1}^Na_m|0\cdots1_m\cdots0\ra=a_1|0\cdots01\ra+a_2|0\cdots10\ra+\cdots+a_N|10\cdots0\ra,
\end{equation}
with $\sum_{m=1}^N|a_m|^2=1$.

For non zero $a_1$, $a_2, \cdots, a_N$, one can show that
$|W_N\ra_g$ is equivalent to $|W_N\ra$ under stochastic local
operation and classical communication (SLOCC) \cite{W. D}, $|W_N
\ra=A_1\otimes A_2\otimes \cdots \otimes A_N|W_N\ra_g$, with
\begin{eqnarray*} A_1= \left(
\begin{array}{cc}
1 & 0  \\
0 & \frac{1}{a_N\sqrt{N}}
\end{array}
\right), ~
A_2= \left(
\begin{array}{cc}
1 & 0  \\
0 & \frac{1}{a_{N-1}\sqrt{N}}
\end{array}
\right),~
 \cdots,~
 A_N= \left(
\begin{array}{cc}
1 & 0  \\
0 & \frac{1}{a_1\sqrt{N}}
\end{array}
\right).
\end{eqnarray*}
Therefore in this case $|W_N\ra_g$ are all genuine $N$-partite entangled states.
Similarly if the number of nonzero coefficients $a_i$ of $|W_N\ra_g$ is $t~(0<t<N)$, then the states are
genuine $t$-partite entangled ones.

Let us consider mixed states with ensembles of pure state
decomposition $\{p_k, |\phi_k\ra\}$, with $|\phi_k \ra =\sum_{m=1}^N
\sqrt{\rho_{mm}}e^{i\theta^{(k)}_m} |0 \cdots 1_m \cdots 0 \ra$,
here $|0\cdots1_m\cdots 0\ra$ denotes a state with the $m$-th
position from right one and others positions zeros, i.e. $|1_N 0
\cdots 0\ra=|10 \cdots 0\ra$, $|01_{(N-1)} \cdots 0\ra=|01 \cdots
0\ra$ and so on. Such states are generally of the form
\begin{equation}\label{rho}
\rho=\sum_{m,n=1}^N \rho_{mn}|0 \cdots 1_m \cdots 0 \ra \la 0 \cdots
1_n \cdots 0|,
\end{equation}
with $\sum_{m=1}^N \rho_{mm} =1$. Such mixed states $\rho$
has only ensemble realizations with
pure states of the form (\ref{gw}).

To study the entanglement property of state (\ref{rho}), we
consider the partial transposition, for instance, with respect to
the $N$-th subsystem, which gives rise to
\begin{eqnarray*} \rho^{PT}= \left(
\begin{array}{ccccccccccc}
0 & 0 & 0 & \rho_{12} & 0 & \cdots & 0 & \cdots & \rho_{1N} & \cdots & 0\\
0 & \rho_{11} & 0 & 0 & 0 & \cdots & 0 & \cdots & 0 & \cdots & 0\\
0 & 0 & \rho_{22} & 0 & \rho_{23} & \cdots & \rho_{2N} & \cdots & 0 & \cdots & 0\\
\rho_{21} & 0 & 0 & 0 & 0 & \cdots & 0 & \cdots & 0 & \cdots & 0\\
0 & 0 & \rho_{32} & 0 & \rho_{33} & \cdots & \rho_{3N} & \cdots & 0 & \cdots & 0\\
\cdots & \cdots & \cdots & \cdots & \cdots & \cdots & \cdots &
\cdots & \cdots & \cdots & \cdots \\
0 & 0 & \rho_{N2} & 0 & \rho_{N3} & \cdots & \rho_{NN} & \cdots & 0 & \cdots & 0\\
\cdots & \cdots & \cdots & \cdots & \cdots & \cdots & \cdots &
\cdots & \cdots & \cdots & \cdots \\
\rho_{N1} & 0 & 0 & 0 & 0 & \cdots & 0 & \cdots & 0 & \cdots & 0\\
\cdots & \cdots & \cdots & \cdots & \cdots & \cdots & \cdots &
\cdots & \cdots & \cdots & \cdots \\
0 & 0 & 0 & 0 & 0 & \cdots & 0 & \cdots & 0 & \cdots & 0\\
\end{array}
\right).
\end{eqnarray*}
By carrying out some elementary transformations, $\rho^{PT}$ can be
transformed into another matrix $(\rho^{PT})^\prime$:
\begin{eqnarray*} (\rho^{PT})^\prime= \left(
\begin{array}{cccc}
A & 0 & 0 & 0 \\
0 & \rho_{11} & 0 & 0 \\
0 & 0 & C & 0 \\
0 & 0 & 0 & D \\
\end{array}
\right),
\end{eqnarray*}
where
\begin{eqnarray*} A= \left(
\begin{array}{cccc}
0 & \rho_{12} & \cdots & \rho_{1N} \\
\rho_{21} & 0 & \cdots & 0 \\
\cdots & \cdots & \cdots & \cdots \\
\rho_{N1} & 0 & \cdots & 0 \\
\end{array}
\right),~~~
 C= \left(
\begin{array}{cccc}
\rho_{22} & \rho_{23} & \cdots & \rho_{2N} \\
\rho_{32} & \rho_{33} & \cdots & \rho_{3N} \\
\cdots & \cdots & \cdots & \cdots \\
\rho_{N2} & \rho_{N3} & \cdots & \rho_{NN} \\
\end{array}
\right),
\end{eqnarray*}
and $D$ is a zero matrix. $C$ is positive semidefinite as $\rho$ is
a density matrix. Therefore the positivity of the matrix
$(\rho^{PT})^\prime$, and hence $(\rho^{PT})$, depends on the matrix
$A$. By deduction we get that the eigenvalues of $A$ are
$\pm(\sum_{j\neq1}|\rho_{1j}|^2)^{\frac{1}{2}}$ and 0. Hence
$\rho^{PT}$ is positive semidefinite if and only if
$(\sum_{j\neq1}|\rho_{1j}|^2)^{\frac{1}{2}}=0$, that is, $\rho_{1j}
=0 $ for $j=2, \cdots, N$. In this case the mixed state (\ref{rho})
becomes
$$
\rho=\rho_{11}|0\cdots0\ra
\la0\cdots0| \otimes |1\ra
\la1|+\sum_{m,n=2}^{N}\rho_{mn}|0\cdots1_m\cdots0\ra
\la0\cdots1_n\cdots0| \otimes |0\ra \la0|.
$$
Therefore it is a bi-separable state with respect to partition $12\cdots N-1$ and $N$ subsystems.
Similar results can be obtained related to partial transpositions with respect to other
subsystems. We have

{\bf Proposition.} The mixed state (\ref{rho}) is a bi-separable one
with respect to partition the $i$-th system and the rest systems if
and only if it is positive semidefinite under partial transposition
with respect to the $i$-th subsystem.

The above property is rather special compared with the ones of SC
states. Moreover, the entanglement of the state $\rho$ is very
robust against particle loss. As the state $|W_N\ra$ remains
entangled even if any $N-2$ parties lose the information, any two
out of $N$ parties possess an entangled state, independent of
whether the remaining $N-2$ parties decide to cooperate with them or
not. Therefore if the mixed state (\ref{rho}) is genuine entangled,
the reduced density matrix of $\rho$, for instance,
$\rho_{\overline{N}}=tr_N
\rho=\rho_{11}|0\cdots0\ra\la0\cdots0|+\sum_{m,n=2}^{N}\rho_{mn}|0\cdots1_m\cdots0\ra
\la0\cdots1_n\cdots0|$ is still a genuine entangled state. The SC
states have no such property. Any kinds of reduced density matrices
of $\rho_{SC}$ states are fully separable. For two dimensional $N$
partite $\rho_{SC}$ state, it is fully separable if and only if
$\rho_{SC}=|\psi\ra$, $|\psi\ra = |0\cdots 0\ra$ or $|\psi\ra =
|1\cdots 1\ra$. But the generalized mixed $W$ state $\rho$
(\ref{rho}) is fully separable if and only if $\rho=\sum_i \rho_{ii}
|0\cdots 1_i \cdots 0\ra \la 0\cdots 1_i \cdots 0 |$, which is
different from SC states.

\section{Monogamy relation of the Generalized mixed W states in multiqubits system}

We now study some monogamy relations related to the generalized
mixed W states in multiqubits system. Recall that the concurrence of
any bipartite pure state $|\psi\ra$ in system
$\mathcal{H}_A\otimes\mathcal{H}_B$ is defined as
$C(|\psi\ra)=\sqrt{2(1-tr\rho_A^2)}$, where
$\rho_A=tr_B|\psi\ra\la\psi|$. The concurrence is then extended to
mixed states $\rho$ by the convex roof:
$C(\rho)\equiv\min_{\{p_i,|\psi_i\ra\}}\sum_ip_iC(|\psi_i\ra)$ for
all possible ensemble realizations
$\rho=\sum_ip_i|\psi_i\ra\la\psi_i|$, where $p_i\geq0$ and
$\sum_ip_i=1$. For a pure three-qubit state $|\psi_{ABC}\ra$,
Coffman, Kundu and Wootters (CKW) \cite{CKW} introduced a monogamy
inequality in terms of concurrence, $C^2_{AB}+C^2_{AC}\leq
C^2_{A(BC)}$, where $C_{AB}$ and $C_{AC}$ are the concurrences of
the mixed states $\rho_{AB}=tr_C(|\psi_{ABC}\ra\la\psi_{ABC}|)$ and
$\rho_{AC}=tr_B(|\psi_{ABC}\ra\la\psi_{ABC}|)$, respectively, and
$C_{A(BC)}$ is the concurrence of $|\psi_{ABC}\ra$ under bipartite
decomposition between subsystems  $A$ and $BC$. The general monogamy
inequality for the case of $n$ qubits is proved in \cite{T. J.
Osborne}. Ref. \cite{Jeong} provided the general monogamy relation
of $|W_N\ra_g$ state with respect to arbitrary partitions. Recently
another monogamy inequality in terms of negativity is deduced in
\cite{Yong}. Negativity is an entanglement measure in two partite
systems which can be expressed as
$N(\rho)=\frac{\|\rho^{PT}\|-1}{2}$, where PT stands for partial
transposition and the trace norm $\|R\|$ is given by
$\|R\|=tr\sqrt{RR^{\dagger}}$. In fact the negativity of state
$\rho$ is essentially the absolute value of the sum of negative
eigenvalues of $\rho^{PT}$. For any three-qubit pure state
$|\psi\ra_{ABC}$, the following CKW-inequality-like monogamy
inequality in terms of negativity holds,
\begin{eqnarray}\label{negabc}
N^2_{AB}+N^2_{AC}\leq N^2_{A(BC)},
\end{eqnarray}
where $N_{AB}$ and $N_{AC}$ are the negativities of the mixed states
$\rho_{AB}$ and $\rho_{AC}$ respectively. $N_{A(BC)}$ is the
negativity of $|\psi_{ABC}\ra$ for the bipartite partition of
subsystems $A$ and $BC$. Similarly one has also
$$
N^2_{BA}+N^2_{BC}\leq N^2_{B(AC)},~~
N^2_{CA}+N^2_{CB}\leq N^2_{C(AB)}.
$$

The general monogamy relation in terms of negativity is given by
\begin{eqnarray}\label{negabcn}
N^2_{A_1A_2}+N^2_{A_1A_3}+\cdots+N^2_{A_1A_N}\leq
N^2_{A_1(A_2A_3\cdots A_N)}.
\end{eqnarray}
Other general monogamy inequalities corresponding to different focused
subsystems $A_i$ can be written down similar to the form (\ref{negabcn}). In the context of quantum
cryptography, such a monogamy property is of fundamental importance
because it quantifies how much information an eavesdropper could
potentially obtain about the secret key extraction. The constraints
on shareability of entanglement lie also at the heart of the success
of many information-theoretic protocols, such as entanglement
distillation \cite{T. J. Osborne}. In this section we prove the
monogamy relation of the mixed state (\ref{rho}) in terms of
negativity.

From the above section we have that the negativity of $\rho$
for a bipartite decomposition between subsystem $A_i$ and the rest subsystems
(non $A_i$) $\overline{A_i}$ is $(\sum_{j\neq
N+1-i}|\rho_{N+1-i,j}|^2)^{\frac{1}{2}}$. Therefore we get
\begin{eqnarray}\label{nega1n}
N_{A_1(A_2\cdots A_N)}^2=\sum_{j\neq N}|\rho_{Nj}|^2.
\end{eqnarray}
As $\rho_{A_1A_2}=tr_{A_3\cdots
A_N}\rho=(\rho_{11}+\cdots+\rho_{N-2,N-2})|00\ra\la00|+\rho_{N-1,N-1}|01\ra\la01|+
\rho_{N-1,N}|01\ra\la10|+\rho_{N,N-1}|10\ra\la01|+\rho_{NN}|10\ra\la10|$,
we get
\begin{eqnarray*}
N_{A_1A_2}=\frac{1}{2}(\sqrt{(\rho_{11}+\cdots+\rho_{N-2,N-2})^2+
4|\rho_{N-1,N}|^2}-(\rho_{11}+\cdots+\rho_{N-2,N-2})).
\end{eqnarray*}

Similarly we can deduce
\begin{eqnarray*}
N_{A_1A_i}&=&\frac{1}{2}(\sqrt{(\rho_{11}+\cdots+\overline{\rho_{N+1-i,N+1-i}}+\cdots+\rho_{N-1,N-1})^2+
4|\rho_{N+1-i,N}|^2}\\
&&-(\rho_{11}+\cdots+\overline{\rho_{N+1-i,N+1-i}}+\cdots+\rho_{N-1,N-1})),
\end{eqnarray*}
where $\overline{\rho_{N+1-i,N+1-i}}$ means that the term is absent in the summation.
As $\sqrt{a+b}\leq\sqrt{a}+\sqrt{b}$, we have
\begin{eqnarray}\label{nega1i}
N_{A_1A_i}&=&\frac{1}{2}(\sqrt{(\rho_{11}+\cdots+\overline{\rho_{N+1-i,N+1-i}}+\cdots+\rho_{N-1,N-1})^2+
4|\rho_{N+1-i,N}|^2}\\\nonumber
&&-(\rho_{11}+\cdots+\overline{\rho_{N+1-i,N+1-i}}+\cdots+\rho_{N-1,N-1}))\\\nonumber
&\leq& |\rho_{N+1-i,N}|.
\end{eqnarray}
From (\ref{nega1n}) and (\ref{nega1i}) we see that (\ref{negabcn})
holds for the mixed state $\rho$. Inequality (\ref{nega1i}) becomes
equality if and only if
$\rho_{11}=\cdots=\overline{\rho_{N+1-i,N+1-i}}=\cdots=\rho_{N-1,N-1}=0$
or $\rho_{N+1-i,N}=0$, $i=2,\cdots, N$. Hence one arrives at that if
the inequality (\ref{negabcn}) becomes equality, then $\rho$ is at
least bipartite $A_i|\overline{A_i}$ separable for some $i$, $1\leq
i\leq N$. In other words, monogamy inequality (\ref{negabcn}) holds
strictly for genuine entangled state (\ref{rho}).

Similarly we have
\begin{eqnarray}\label{nega1a2i}
N_{A_kA_l(A_1\cdots \overline{A_k}\cdots\overline{A_l}\cdots
A_N)}^2=\sum_{i=N+1-k, N+1-l,~j\neq N+1-k,N+1-l}|\rho_{ij}|^2.
\end{eqnarray}
For $A_1A_2|A_3\cdots A_N$ partition, the following equalities
holds:
\begin{eqnarray}
N^2_{A_1A_2(A_3\cdots A_N)}&=&\sum_{i=N-1, N,~j\neq
N-1,N}|\rho_{ij}|^2,\\\label{nega1a2an} N_{A_1A_2(A_k)}
&=&\frac{1}{2}(((\rho_{11}+\cdots+\overline{\rho_{N+1-k,N+1-k}}+\cdots+\rho_{N-2,N-2})^2\\\nonumber
&&+4(|\rho_{N+1-k,N-1}|^2
+|\rho_{N+1-k,N}|^2))^{\frac{1}{2}}\\\nonumber
&&-(\rho_{11}+\cdots+\overline{\rho_{N+1-k,N+1-k}}+\cdots+\rho_{N-2,N-2})).\label{nega1a2ai}
\end{eqnarray}
Therefore one gets
\begin{eqnarray}\label{nega1a2ineq}
N^2_{A_1A_2(A_3)}+\cdots+N^2_{A_1A_2(A_N)}\leq N^2_{A_1A_2(A_3\cdots
A_N)}.
\end{eqnarray}
If inequality (\ref{nega1a2ineq}) becomes an equality, then state
$\rho$ is at least separable under some partition
$A_i|A_j|\overline{A_i}\overline{A_j}$, $1\leq i<j\leq N$, otherwise
it will be a strictly inequality. Generally for any partition
$P_1=\{A_{i_1},\cdots,A_{i_k}\}$,
$P_2=\{A_{i_{k+1}},\cdots,A_{i_{k+l}}\}$, $\cdots$,
$P_s=\{A_{i_{k+s}},\cdots,A_{i_N}\}$, we have
\begin{eqnarray}\label{neganyparti}
N^2_{P_1 P_2}+\cdots+N^2_{P_1 P_s}\leq N^2_{P_1(P_2\cdots P_s)}.
\end{eqnarray}
If this inequality becomes an equality, the state is at least
separable under some partition $A_{t_1}|A_{t_2}|\cdots
|A_{t_k}|\overline{A_{t_1}A_{t_2}\cdots A_{t_k}}$, $1\leq
t_1<t_2<\cdots<t_k\leq N$. On the other hand, we can deduce another
conclusion that mixed state $\rho$ (\ref{rho}) is biseparable if and
only if it is PPT with respect to such partition.

\section{COMMENTS AND CONCLUSIONS}

The results can be generalized to $N$-partite
$d$ dimensional systems. That is
\begin{eqnarray}\label{rho2}
\rho=\sum_{i,k=1}^N\sum_{j,l=0}^{d-1}\rho_{i(j),k(l)}|0\cdots
j_i\cdots0\ra\la0\cdots l_k\cdots0|
\end{eqnarray}
with $\sum_{i=1}^N\sum_{j=0}^{d-1}\rho_{i(j),i(j)}=1$.
We can similarly prove that pure state decomposition of (\ref{rho2}) has the form
$|\psi\ra=\sum_{i=1}^N\sum_{j=0}^{d-1}a_{i(j)}|0\cdots
j_i\cdots0\ra$, which is equivalent to pure state
$|\psi\ra=\sum_{i=1}^N\sum_{j=0}^{d-1}|0\cdots j_i\cdots0\ra$
under SLOCC. Moreover state (\ref{rho2}) is separable with respect to the subsystem $A_i$ $(1\leq
i\leq N)$ if and only if (\ref{rho2}) is PPT with respect to $A_i$.

The monogamy relations can be similarly studied. For example, the
negative eigenvalue of $\rho^{PT}$ with respect to the first subsystem is
$(\sum_{i\neq
N}\sum_{j,l=0}^{d-1}|\rho_{i(j),N(l)}|^2)^{\frac{1}{2}}$. Therefore
the negativity $N^2_{A_1(A_2\cdots A_N)}=\sum_{i\neq
N}\sum_{j,l=0}^{d-1}|\rho_{i(j),N(l)}|^2$. By tedious calculation we
can show that inequality (\ref{negabcn}) and (\ref{neganyparti})
also hold for state (\ref{rho2}). And if the inequalities become
equalities, then corresponding results hold similar to qubit case. While for
SC states, the equalities hold if and only if they are fully
separable, as their reduced matrices are all fully separable.

In summary, similar to SC states having generalized GHZ states in
the pure state decomposition, we have introduced a new class of
multipartite entangled mixed states with pure state decompositions
of generalized $W$ states. The entanglement and separability
properties are studied according to PPT operations. It is shown that
the states are bipartite separable if they are PPT  corresponding
such partition. Monogamy relations related to these states are also
investigated. Although it is still not known if the monogamy
relations in terms of negativity hold for general high dimensional
mixed state, they are true for our class of states. Above all, the
entanglement of these states is very robust against particle loss.
If the mixed state (\ref{rho}) is genuine entangled, the reduced
density matrix of $\rho$ is still a genuine entangled state.