Multi-task Learning for Cross-Lingual Sentiment Analysis
Thakkar, Gaurish
This paper presents a cross-lingual sentiment analysis of news articles using zero-shot and few-shot learning. The study aims to classify the Croatian news articles with the positive, negative, and neutral sentiment using the Slovene dataset. The system is based on a trilingual BERT-based model trained in three languages: English, Slovene,Croatian. The paper analyses different setups of using datasets in two languages and proposes a simple multi-task model to perform sentiment classification. The evaluation is performed using the few-shot and zero-shot scenarios in single-task and multi-task experiments for Croatian and Slovene.
 Yeah, so hello everyone and I'm back. So I'll be presenting our paper with my mentor, Nevis Mikhailovich Przaryovic and Marko Tadich that is multi-task learning for cross-lingual sentiment analysis. Okay, so it will be a short presentation with the introduction, the methodology, the experimental setup, the results and the conclusion. So as we know, sentiment analysis is the field of study of analysis of people's opinions, sentiments, appraisals, attitudes, emotions towards the entities and they are in their express written texts. So usually we see sentiment analysis being well studied in English language, wherein we have a large amount of training data and when you test. So in cross-lingual study, so this paper deals with cross-lingual sentiment analysis of Croatian, wherein practically we are not using English training data, but we would leverage a dataset from another language which is Slovene to kind of perform a cross-lingual sentiment transfer. Reason why we chose Slovene, reason being the mutual intangibility between the Slavic languages that was a paper by Elena, which showed that in Indian life Croatian and Slovene have really high mutual intangibility, which means the speaker does not require a really large amount of effort to understand the other. So we thought maybe this also applies to the language transfer that could be possible. So we have two datasets, luckily we have one in Croatian which is like 2000 documents only. So these are Croatian news articles tagged at document level and then we have Slovene news articles which is a really big dataset wherein we have like 10,000 documents, they are tagged at three different levels. Each of the distributions is distributed in the table, like we can see in the table and it's a massive dataset. So coming on to the next, we have a really simple methodology wherein what we do is we have a shared encoder wherein we have three different heads being trained on three different labels from each of the datasets. So the sent in news has document levels, paragraph level and the same sentence is being tagged. So we have three different levels for that. But since we assume the mutual intangibility would play a role, so we have Croatian document being combined with Slovene document and being fed into the shared encoder and then being trained with the respective tasks wherein we cycle through each of these tasks one by one randomly during the training. So we have around five different experiments that we performed with this dataset. So we have first single task Slovene wherein we train only with Slovene document level data and then we test it on Croatian document. Then we have multitask Slovene wherein we use all the three levels from the Slovene data and we do not use Croatian data in these two previous settings, but we do test it on the Croatian data. Then we plug in only the Croatian level document and then we try to see how does it work. This is like the normal classic fine tuning of a transformer. It's not that different. And the last wherein we have a multitask model wherein we combine Croatian and Slovene at the document level, but we also train the other compatible heads with their respective datasets that is paragraph and sentence. And the last but not the least, we'd also try to see what happens when we take away the paragraph and the sentence and just train with document and compare the overall performance. So we do see that the datasets are a bit noisy. So we perform a little bit of reprocessing. We remove the empty strings and perform the deduplication which does reduce the size of the dataset. Then we perform 80-20 split and then keep like 10% for the Dev set. One thing that we changed from a typical language model, mass language model scenario would be we use a bird-based language model called close slow angle which is trained with Slovene Croatian and English. Reason being this model was, it has been shown that this model, this encoder performs well for the task of parse and dependency parsing with respect to M-bird. But it substantially outperforms the M-bird when it comes to these three tasks. So we have like three different labels, positive negative neutral and we train it with the batch size of 32. This worthy results as a simple baseline we use just a majority class. Then coming on to single task, Slovene, we see that only using document, this worthy scores. The single task, oh sorry, Slovene multitask performs well on Slovene but when we do a zero shot on the Croatian, the scores are, the F1 is kind of 50. But putting in the Croatian model itself, obviously it's like the classical fine tuning case, we have F1 of 56. But then coming on to when we start adding Croatian data to the Slovene, I mean we do see a drop in document level for the, for a Slovene. But then we start seeing an increase in the Croatian and finally for the multitask when we combine the document of Slovene and the Croatian model, there is a slight decrease with compared to the Slovene multitask. But overall we have a better performance with respect to precision recall for the Croatian classification. Coming to the discussion, I mean I have already said that the multitask setup performs better than the other settings. The best document model for the Slovene was Slovene itself, but it performs poor when we add Croatian document and the worst performing Croatian model was when we use only Slovene with the single task. Nevertheless, it performs better with a Slovene set with respect to conclusion. We present our simple setup for the cross-lingual sentimentals for Croatian and we utilized the really large Slovene dataset in order to improve the final cross-lingual transfer. Obviously in order to check this in a, since Slovene is also under resource language, we would like to see how can we utilize English along with Croatian and Slovene that is our ongoing work. Also in the same way as we saw that this dataset has, this dataset, the Croatian descendants level doesn't exist. So we have approximately finished our work on that. Maybe soon we will have one paper wherein we compare the levels with Slovene and Croatian. And coming to the error analysis, we also found that the words do play a major role, like something that has, so the model tries to capture like positive words even though it could be neutral news, but that it fails major majority on these words. And we need to check that like how can we improve the performance on that. Our code is available on the link and I'm open for questions.
 Yeah, so hello everyone and I'm back. So I'll be presenting our paper with my mentor,  Nevis Mikhailovich Przaryovic and Marko Tadich that is multi-task learning for cross-lingual  sentiment analysis. Okay, so it will be a short presentation with the introduction, the methodology,  the experimental setup, the results and the conclusion. So as we know, sentiment analysis  is the field of study of analysis of people's opinions, sentiments, appraisals, attitudes,  emotions towards the entities and they are in their express written texts. So usually we see  sentiment analysis being well studied in English language, wherein we have a large  amount of training data and when you test. So in cross-lingual study, so this paper deals with  cross-lingual sentiment analysis of Croatian, wherein practically we are not using English  training data, but we would leverage a dataset from another language which is Slovene to kind  of perform a cross-lingual sentiment transfer. Reason why we chose Slovene, reason being the  mutual intangibility between the Slavic languages that was a paper by Elena, which showed that  in Indian life Croatian and Slovene have really high mutual intangibility, which means the  speaker does not require a really large amount of effort to understand the other. So we thought  maybe this also applies to the language transfer that could be possible. So we have two datasets,  luckily we have one in Croatian which is like 2000 documents only. So these are Croatian news  articles tagged at document level and then we have Slovene news articles which is a really big  dataset wherein we have like 10,000 documents, they are tagged at three different levels.  Each of the distributions is distributed in the table, like we can see in the table and  it's a massive dataset. So coming on to the next, we have a really simple methodology wherein what  we do is we have a shared encoder wherein we have three different heads being trained on  three different labels from each of the datasets. So the sent in news has document levels, paragraph  level and the same sentence is being tagged. So we have three different levels for that. But  since we assume the mutual intangibility would play a role, so we have Croatian document being  combined with Slovene document and being fed into the shared encoder and then being trained with  the respective tasks wherein we cycle through each of these tasks one by one randomly during  the training. So we have around five different experiments that we performed with this dataset.  So we have first single task Slovene wherein we train only with Slovene document level data and  then we test it on Croatian document. Then we have multitask Slovene wherein we use all the three  levels from the Slovene data and we do not use Croatian data in these two previous settings,  but we do test it on the Croatian data. Then we plug in only the Croatian level document and then  we try to see how does it work. This is like the normal classic fine tuning of a transformer.  It's not that different. And the last wherein we have a multitask model wherein we combine Croatian  and Slovene at the document level, but we also train the other compatible heads with their respective  datasets that is paragraph and sentence. And the last but not the least, we'd also try to see what  happens when we take away the paragraph and the sentence and just train with document and compare  the overall performance. So we do see that the datasets are a bit noisy. So we perform a little  bit of reprocessing. We remove the empty strings and perform the deduplication which does reduce  the size of the dataset. Then we perform 80-20 split and then keep like 10% for the Dev set.  One thing that we changed from a typical language model, mass language model scenario would be  we use a bird-based language model called close slow angle which is trained with Slovene Croatian  and English. Reason being this model was, it has been shown that this model, this encoder performs  well for the task of parse and dependency parsing with respect to M-bird. But it substantially  outperforms the M-bird when it comes to these three tasks. So we have like three different labels,  positive negative neutral and we train it with the batch size of 32. This worthy results as a  simple baseline we use just a majority class. Then coming on to single task, Slovene, we see that  only using document, this worthy scores. The single task, oh sorry, Slovene multitask  performs well on Slovene but when we do a zero shot on the Croatian, the scores are,  the F1 is kind of 50. But putting in the Croatian model itself, obviously it's like the classical  fine tuning case, we have F1 of 56. But then coming on to when we start adding Croatian data to the  Slovene, I mean we do see a drop in document level for the, for a Slovene. But then we start seeing  an increase in the Croatian and finally for the multitask when we combine the document of  Slovene and the Croatian model, there is a slight decrease with compared to the Slovene multitask.  But overall we have a better performance with respect to precision recall for the Croatian  classification. Coming to the discussion, I mean I have already said that the multitask setup  performs better than the other settings. The best document model for the Slovene was Slovene itself,  but it performs poor when we add Croatian document and the worst performing Croatian model was when  we use only Slovene with the single task. Nevertheless, it performs better with a Slovene set  with respect to conclusion. We present our simple setup for the cross-lingual sentimentals for  Croatian and we utilized the really large Slovene dataset in order to improve the final cross-lingual  transfer. Obviously in order to check this in a, since Slovene is also under resource language,  we would like to see how can we utilize English along with Croatian and Slovene that is our  ongoing work. Also in the same way as we saw that this dataset has, this dataset, the Croatian  descendants level doesn't exist. So we have approximately finished our work on that. Maybe  soon we will have one paper wherein we compare the levels with Slovene and Croatian.  And coming to the error analysis, we also found that the words do play a major role,  like something that has, so the model tries to capture like positive words even though it could  be neutral news, but that it fails major majority on these words. And we need to  check that like how can we improve the performance on that.  Our code is available on the link and I'm open for questions.
Gaurish_Thakkar_2_0_371.png Gaurish_Thakkar_2_372_623.png Gaurish_Thakkar_2_624_2567.png Gaurish_Thakkar_2_2568_3395.png Gaurish_Thakkar_2_3396_4619.png Gaurish_Thakkar_2_4620_6419.png Gaurish_Thakkar_2_6420_8111.png Gaurish_Thakkar_2_8112_10247.png Gaurish_Thakkar_2_10248_10967.png Gaurish_Thakkar_2_10968_11975.png Gaurish_Thakkar_2_11976_12023.png Gaurish_Thakkar_2_12024_12287.png Gaurish_Thakkar_2_12288_12455.png Gaurish_Thakkar_2_12456_13343.png Gaurish_Thakkar_2_13344_13501.png
Yeah, so hello everyone and I'm back. So I'll be presenting our paper with my mentor, Nevis Mikhailovich Przaryovic and Marko Tadich that is multi-task learning for cross-lingual sentiment analysis. Okay, so it will be a short presentation with the introduction, the methodology, the experimental setup, the results and the conclusion. So as we know, sentiment analysis is the field of study of analysis of people's opinions, sentiments, appraisals, attitudes, emotions towards the entities and they are in their express written texts. So usually we see sentiment analysis being well studied in English language, wherein we have a large amount of training data and when you test. So in cross-lingual study, so this paper deals with cross-lingual sentiment analysis of Croatian, wherein practically we are not using English training data, but we would leverage a dataset from another language which is Slovene to kind of perform a cross-lingual sentiment transfer. Reason why we chose Slovene, reason being the mutual intangibility between the Slavic languages that was a paper by Elena, which showed that in Indian life Croatian and Slovene have really high mutual intangibility, which means the speaker does not require a really large amount of effort to understand the other. So we thought maybe this also applies to the language transfer that could be possible. So we have two datasets, luckily we have one in Croatian which is like 2000 documents only. So these are Croatian news articles tagged at document level and then we have Slovene news articles which is a really big dataset wherein we have like 10,000 documents, they are tagged at three different levels. Each of the distributions is distributed in the table, like we can see in the table and it's a massive dataset. So coming on to the next, we have a really simple methodology wherein what we do is we have a shared encoder wherein we have three different heads being trained on three different labels from each of the datasets. So the sent in news has document levels, paragraph level and the same sentence is being tagged. So we have three different levels for that. But since we assume the mutual intangibility would play a role, so we have Croatian document being combined with Slovene document and being fed into the shared encoder and then being trained with the respective tasks wherein we cycle through each of these tasks one by one randomly during the training. So we have around five different experiments that we performed with this dataset. So we have first single task Slovene wherein we train only with Slovene document level data and then we test it on Croatian document. Then we have multitask Slovene wherein we use all the three levels from the Slovene data and we do not use Croatian data in these two previous settings, but we do test it on the Croatian data. Then we plug in only the Croatian level document and then we try to see how does it work. This is like the normal classic fine tuning of a transformer. It's not that different. And the last wherein we have a multitask model wherein we combine Croatian and Slovene at the document level, but we also train the other compatible heads with their respective datasets that is paragraph and sentence. And the last but not the least, we'd also try to see what happens when we take away the paragraph and the sentence and just train with document and compare the overall performance. So we do see that the datasets are a bit noisy. So we perform a little bit of reprocessing. We remove the empty strings and perform the deduplication which does reduce the size of the dataset. Then we perform 80-20 split and then keep like 10% for the Dev set. One thing that we changed from a typical language model, mass language model scenario would be we use a bird-based language model called close slow angle which is trained with Slovene Croatian and English. Reason being this model was, it has been shown that this model, this encoder performs well for the task of parse and dependency parsing with respect to M-bird. But it substantially outperforms the M-bird when it comes to these three tasks. So we have like three different labels, positive negative neutral and we train it with the batch size of 32. This worthy results as a simple baseline we use just a majority class. Then coming on to single task, Slovene, we see that only using document, this worthy scores. The single task, oh sorry, Slovene multitask performs well on Slovene but when we do a zero shot on the Croatian, the scores are, the F1 is kind of 50. But putting in the Croatian model itself, obviously it's like the classical fine tuning case, we have F1 of 56. But then coming on to when we start adding Croatian data to the Slovene, I mean we do see a drop in document level for the, for a Slovene. But then we start seeing an increase in the Croatian and finally for the multitask when we combine the document of Slovene and the Croatian model, there is a slight decrease with compared to the Slovene multitask. But overall we have a better performance with respect to precision recall for the Croatian classification. Coming to the discussion, I mean I have already said that the multitask setup performs better than the other settings. The best document model for the Slovene was Slovene itself, but it performs poor when we add Croatian document and the worst performing Croatian model was when we use only Slovene with the single task. Nevertheless, it performs better with a Slovene set with respect to conclusion. We present our simple setup for the cross-lingual sentimentals for Croatian and we utilized the really large Slovene dataset in order to improve the final cross-lingual transfer. Obviously in order to check this in a, since Slovene is also under resource language, we would like to see how can we utilize English along with Croatian and Slovene that is our ongoing work. Also in the same way as we saw that this dataset has, this dataset, the Croatian descendants level doesn't exist. So we have approximately finished our work on that. Maybe soon we will have one paper wherein we compare the levels with Slovene and Croatian. And coming to the error analysis, we also found that the words do play a major role, like something that has, so the model tries to capture like positive words even though it could be neutral news, but that it fails major majority on these words. And we need to check that like how can we improve the performance on that. Our code is available on the link and I'm open for questions.
multi-task learning for cross-lingual sentiment cross-lingual study, so this paper deals with cross-lingual sentiment analysis of Croatian, wherein practically we are not using English perform a cross-lingual sentiment transfer. only. So these are Croatian news articles tagged heads being trained on three different labels from each of the datasets. So the sent in news experiments that we performed with this dataset. a multitask model wherein we combine Croatian and labels, positive negative neutral and we train it Slovene and the Croatian model, there is a slight decrease with compared to the Slovene multitask.