The Potential of Using Vision Videos for CrowdRE: Video Comments as a Source of Feedback
Karras, Oliver
Vision videos are established for soliciting feedback and stimulating discussions in requirements engineering (RE) practices such as focus groups. Different researchers motivated the transfer of these benefits into crowd-based RE (CrowdRE) by using vision videos on social media platforms. So far, however, little research explored the potential of using vision videos for CrowdRE in detail. In this paper, we analyze and assess this potential, in particular, focusing on video comments as a source of feedback. In a case study, we analyzed 4505 comments on a vision video from YouTube. We found that the video solicited 2770 comments from 2660 viewers in four days. This is more than 50% of all comments the video received in four years. Even though only a certain fraction of these comments are relevant to RE, the relevant comments address typical intentions and topics of user feedback, such as feature request or problem report. Besides the typical user feedback categories, we found more than 300 comments that address the topic safety which has not appeared in previous analyses of user feedback. In an automated analysis, we compared the performance of three machine learning algorithms on classifying the video comments. Despite certain differences, the algorithms classified the video comments well. Based on these findings, we conclude that the use of vision videos for CrowdRE has a large potential. Despite the preliminary nature of the case study, we are optimistic that vision videos can motivate stakeholders to actively participate in a crowd and solicit numerous of video comments as a valuable source of feedback.
 Hello and welcome to my talk with the title, the potential of using vision videos for crowd-area video comments as a source of feedback. My name is Odeva Kavas and I'm working at Latinx Information Center for Science and Technology. Let's start with considering crowd-area and its use of feedback sources. While according to literature, there are some frequently used sources like user forums or mobile application markets, others, and in particular, social media platforms are rarely used. Nevertheless, different researchers agreed on the potential of social media platforms since they can provide new motivation opportunities for more than crowd members, for example, due to entertaining and enjoyable activities such as watching a video. In addition, they provide the opportunity to achieve a wide reach so you can gather millions of views and solicit thousands of comments. We want to further complete it with the idea of using vision videos on social media platforms. For this purpose, we need to understand what is a vision video. A vision video is a video that represents a vision or process for achieving shared understanding among all parties involved by this clothing, discussing and aligning their mental models of the future system. So far, vision videos are frequently used in individual meetings, focus groups, or workshops in order to stimulate discussions and solicit feedback. Researchers proposed the idea to turn fairer disease benefits of vision videos for crowd or E by using them on social media platforms. In this talk, we want to consider the research question, what is the potential of using vision videos on social media platforms to solicit feedback in the form of video comments for crowd or E? We investigated this research question from three perspectives regarding the quantity of views and comments, the content of the comments, and their suitability for analysis used in crowd or E. In particular, we followed a clear research approach consisting of the four main steps, data collection, data cleaning, and manual analysis for collector data, and first, automated analysis. In the manual analysis, we spent a lot of effort in order to identify the different classification categories and create a VNT label data set, which can be used for the automated analysis. In particular, we selected YouTube as a platform and the vision video tunnels by the Warren Company. At the time, we downloaded the data set, the video had 4,500 comments and 2,100 replies. The comments are directly related to the video while the replies address some kind of discussions below the respective comment. For this reason, we focus only on this paper on the end of this subsequent comment. After the data cleaning, the data set contained 4,400 comments for the sub-C. and manual analysis. If you are interested in our data set scripts and the coding guides, the replication packets are provided on the node. Now let's consider some of the findings of the paper. For the perspective of quantity, how that after four days, there were already 2,770 comments. This is interesting, then these are 53% of all comments that were collected in four years. So we can say that a large number of comments was received in a really short amount of time. The result of the content perspective are based on a comprehensive manual analysis to collect the video comments. For this purpose, we had two coders who performed for us two classification tasks of the video comment regarding their relevance and polarity, as well as following up by the intentional topic. After each classification task, we had a discussion meeting to resolve any disagreements between the two coders. We found that a seven times 64 comments are only relevant from the point of view CROT or E. The further investigation of the intentional topics showed that the main intent is a feature request and problem report, which are addressed in the video comments. In addition, we found that the topics efficiency and safety are the most frequent topics addressed on the video comments. For the classification regarding the intentional topic, we used the taxonomy of user classification provided by Santa Cetari. What we found is that a lot of the content in the comments is related to the taxonomy of user feedback classification categories. Regarding the last perspective of suitability, we investigated how well the video comments can be classified by machine learning algorithms that are frequently applied in CROT or E. Our selection is based on the overview of user feedback classification approaches provided by Santa Cetari. We mainly use three algorithms and two feature. These algorithms support vector machines and they are based at the most frequently used ones in CROT or E. And in addition, we focus on the use of random forest which is rarely used but showed already good performance in different studies. For the features, we focused on back-of-words and TF-IDF. In particular, we created for each category a balanced stage set and performed a 10-fold cross validation for binary classification task. To reduce data splitting bias, we repeated each 10-fold cross-validation 10 times for each category. Our results show that SVM combined with TF-IDF achieved the best results overall. In particular, we found that the relevant comments and the topic of safety can be classified very well. However, the intention of problem report is the most difficult one to classify. Now we would like to discuss some of our findings. First of all, considering the feedback on safety. If we consider the taxonomy of user feedback classification, we see that it was developed with a systematic literature review on the topic of classifying user feedback. In particular, the authors investigated 43 different papers and identified 78 different categories for user feedback. However, safety is not an element of the taxonomy. And we assume that this lack of safety in the taxonomy is a result of the currently and frequently used user feedback sources, mainly user forums and mobile application markets. Vision videos are in particular concrete by showing the system and its use. From this way, they can empower stakeholders to experience and report their suspected quality news problems, such as economic health or safety risk, even if they have never actually used the system. This benefit is already known from applying videos in focus groups or workshops, for example. This observation and explanation let us conclude that we can see directly that one of the most important benefits of vision videos can be transferred on social media platforms, resulting in a currently completely new and even missing aspect of category in the taxonomy of user feedback classification. As a second aspect, we would like to discuss the selected vision media. We selected the vision media tunnels by the Boring Company, with company responsible by Elon Musk, and in this way, it already has a strong social media presence revolting in thousands of companies. However, this does not vote for average company. Therefore, we need to take strategy how we can build social media presence for companies in order to solicit thousands of video comments. Nevertheless, we need even more besides the social media presence, we also need to bridge the gap to the development. In this way, there's the need for holistic approaches for crowd-re, covering the entire software development process in order to operationally involve the crowd. As an answer to our research question, we can summarize that there's a potential of using vision videos on social media platforms. Regarding the number of commentant there are content with steep promising results for crowd-re. In this way, a vision videos can be suitable option to motivate in crowd numbers to actively participate by writing comments which are available source of feedback. Finally, I would like to conclude the presentation. What we've seen, we've considered the elicitation and the analysis of view of feedback in crowd-re. We've seen that frequently used sources are mobile application market in user forms. However, there are other sources and in particular social media platforms which are rarely used. We've considered the idea of using videos on social media platforms to solicit thousands of comments. We investigated the content of these comments and found interesting result which are promising regarding the use of vision videos for crowd-re. However, there's a lot of future work to do. First of all, there's new trend and text classification using deep learning algorithm. We would suggest to apply these algorithms also on our dataset to assess its usability for crowd-re. In addition, we need other replication and extensions of our study, including the development of first datasets. Due to the lack of suitable datasets, we spent a lot of effort in creating our labeled dataset which hopefully can be further used by other researchers. At the last point, we want to emphasize the need to investigate also the replies to cycle comments on the video. Based on our manual review, we saw that these applies do not address the video itself but show some kind of discussion below the associated comment. We would suggest to use, for example, frameworks like crowd-arrow proposed by Carnet R in order to identify arguments for or against a given statement. In this way, we could obtain further inflight into the source of the crowd. In this way, we want now to close our talk and thank you for your interest. Have a nice day.
 Hello and welcome to my talk with the title,  the potential of using vision videos for  crowd-area video comments as a source of feedback.  My name is Odeva Kavas and I'm working  at Latinx Information Center for Science and Technology.  Let's start with considering  crowd-area and its use of feedback sources.  While according to literature,  there are some frequently used sources like  user forums or mobile application markets,  others, and in particular,  social media platforms are rarely used.  Nevertheless, different researchers  agreed on the potential of social media platforms  since they can provide new motivation opportunities  for more than crowd members,  for example, due to entertaining and enjoyable activities  such as watching a video.  In addition, they provide the opportunity to achieve  a wide reach so you can gather  millions of views and solicit thousands of comments.  We want to further complete it with the idea of  using vision videos on social media platforms.  For this purpose, we need to understand  what is a vision video.  A vision video is a video that represents  a vision or process for achieving shared understanding  among all parties involved by this clothing,  discussing and aligning their mental models  of the future system.  So far, vision videos are frequently used  in individual meetings, focus groups, or workshops  in order to stimulate discussions and solicit feedback.  Researchers proposed the idea to turn  fairer disease benefits of vision videos  for crowd or E by using them on social media platforms.  In this talk, we want to consider the research question,  what is the potential of using vision videos  on social media platforms to solicit feedback  in the form of video comments for crowd or E?  We investigated this research question  from three perspectives regarding the quantity  of views and comments, the content of the comments,  and their suitability for analysis used in crowd or E.  In particular, we followed a clear research approach  consisting of the four main steps, data collection,  data cleaning, and manual analysis for collector data,  and first, automated analysis.  In the manual analysis, we spent a lot of effort  in order to identify the different classification categories  and create a VNT label data set,  which can be used for the automated analysis.  In particular, we selected YouTube as a platform  and the vision video tunnels by the Warren Company.  At the time, we downloaded the data set,  the video had 4,500 comments and 2,100 replies.  The comments are directly related to the video  while the replies address some kind of discussions  below the respective comment.  For this reason, we focus only on this paper  on the end of this subsequent comment.  After the data cleaning, the data set  contained 4,400 comments for the sub-C.  and manual analysis.  If you are interested in our data set scripts  and the coding guides,  the replication packets are provided on the node.  Now let's consider some of the findings of the paper.  For the perspective of quantity,  how that after four days, there were already 2,770 comments.  This is interesting, then these are 53%  of all comments that were collected in four years.  So we can say that a large number of comments  was received in a really short amount of time.  The result of the content perspective  are based on a comprehensive manual analysis  to collect the video comments.  For this purpose, we had two coders who performed  for us two classification tasks of the video comment  regarding their relevance and polarity,  as well as following up by the intentional topic.  After each classification task,  we had a discussion meeting to resolve any disagreements  between the two coders.  We found that a seven times  64 comments are only relevant from the point of view  CROT or E.  The further investigation of the intentional topics  showed that the main intent is a feature request  and problem report,  which are addressed in the video comments.  In addition, we found that the topics efficiency and safety  are the most frequent topics addressed on the video comments.  For the classification regarding the intentional topic,  we used the taxonomy of user classification  provided by Santa Cetari.  What we found is that a lot of the content in the comments  is related to the taxonomy  of user feedback classification categories.  Regarding the last perspective of suitability,  we investigated how well the video comments  can be classified by machine learning algorithms  that are frequently applied in CROT or E.  Our selection is based on the overview  of user feedback classification approaches  provided by Santa Cetari.  We mainly use three algorithms and two feature.  These algorithms support vector machines  and they are based at the most frequently used ones  in CROT or E.  And in addition, we focus on the use of random forest  which is rarely used but showed already good performance  in different studies.  For the features, we focused on back-of-words and TF-IDF.  In particular, we created for each category  a balanced stage set and performed a 10-fold cross validation  for binary classification task.  To reduce data splitting bias,  we repeated each 10-fold cross-validation  10 times for each category.  Our results show that SVM combined with TF-IDF  achieved the best results overall.  In particular, we found that the relevant comments  and the topic of safety can be classified very well.  However, the intention of problem report  is the most difficult one to classify.  Now we would like to discuss some of our findings.  First of all, considering the feedback on safety.  If we consider the taxonomy of user feedback classification,  we see that it was developed with a systematic literature review  on the topic of classifying user feedback.  In particular, the authors investigated 43 different papers  and identified 78 different categories for user feedback.  However, safety is not an element of the taxonomy.  And we assume that this lack of safety in the taxonomy  is a result of the currently and frequently used  user feedback sources, mainly user forums  and mobile application markets.  Vision videos are in particular concrete  by showing the system and its use.  From this way, they can empower stakeholders to experience  and report their suspected quality news problems,  such as economic health or safety risk,  even if they have never actually used the system.  This benefit is already known from applying videos  in focus groups or workshops, for example.  This observation and explanation let us conclude  that we can see directly that one of the most important  benefits of vision videos can be transferred  on social media platforms,  resulting in a currently completely new  and even missing aspect of category  in the taxonomy of user feedback classification.  As a second aspect, we would like to discuss  the selected vision media.  We selected the vision media tunnels  by the Boring Company,  with company responsible by Elon Musk,  and in this way, it already has a strong social media presence  revolting in thousands of companies.  However, this does not vote for average company.  Therefore, we need to take strategy  how we can build social media presence for companies  in order to solicit thousands of video comments.  Nevertheless, we need even more  besides the social media presence,  we also need to bridge the gap to the development.  In this way, there's the need for holistic approaches  for crowd-re, covering the entire software development process  in order to operationally involve the crowd.  As an answer to our research question,  we can summarize that there's a potential  of using vision videos on social media platforms.  Regarding the number of commentant there are content  with steep promising results for crowd-re.  In this way, a vision videos can be suitable option  to motivate in crowd numbers to actively participate  by writing comments which are available source of feedback.  Finally, I would like to conclude the presentation.  What we've seen, we've considered the elicitation  and the analysis of view of feedback in crowd-re.  We've seen that frequently used sources  are mobile application market in user forms.  However, there are other sources  and in particular social media platforms  which are rarely used.  We've considered the idea of using videos  on social media platforms to solicit thousands of comments.  We investigated the content of these comments  and found interesting result which are promising  regarding the use of vision videos for crowd-re.  However, there's a lot of future work to do.  First of all, there's new trend and text classification  using deep learning algorithm.  We would suggest to apply these algorithms  also on our dataset to assess its usability  for crowd-re.  In addition, we need other replication  and extensions of our study,  including the development of first datasets.  Due to the lack of suitable datasets,  we spent a lot of effort in creating our labeled dataset  which hopefully can be further used by other researchers.  At the last point, we want to emphasize  the need to investigate also the replies  to cycle comments on the video.  Based on our manual review,  we saw that these applies do not address the video  itself but show some kind of discussion  below the associated comment.  We would suggest to use, for example,  frameworks like crowd-arrow proposed by Carnet R  in order to identify arguments for or against  a given statement.  In this way, we could obtain further inflight  into the source of the crowd.  In this way, we want now to close our talk  and thank you for your interest.  Have a nice day.
CrowdRE21-Video_Comments_0_323.png CrowdRE21-Video_Comments_324_1271.png CrowdRE21-Video_Comments_1272_2219.png CrowdRE21-Video_Comments_2220_3467.png CrowdRE21-Video_Comments_3468_4415.png CrowdRE21-Video_Comments_4416_5015.png CrowdRE21-Video_Comments_5016_6515.png CrowdRE21-Video_Comments_6516_8411.png CrowdRE21-Video_Comments_8412_10463.png CrowdRE21-Video_Comments_10464_10523.png CrowdRE21-Video_Comments_10524_12155.png CrowdRE21-Video_Comments_12156_14800.png
Hello and welcome to my talk with the title, the potential of using vision videos for crowd-area video comments as a source of feedback. My name is Odeva Kavas and I'm working at Latinx Information Center for Science and Technology. Let's start with considering crowd-area and its use of feedback sources. While according to literature, there are some frequently used sources like user forums or mobile application markets, others, and in particular, social media platforms are rarely used. Nevertheless, different researchers agreed on the potential of social media platforms since they can provide new motivation opportunities for more than crowd members, for example, due to entertaining and enjoyable activities such as watching a video. In addition, they provide the opportunity to achieve a wide reach so you can gather millions of views and solicit thousands of comments. We want to further complete it with the idea of using vision videos on social media platforms. For this purpose, we need to understand what is a vision video. A vision video is a video that represents a vision or process for achieving shared understanding among all parties involved by this clothing, discussing and aligning their mental models of the future system. So far, vision videos are frequently used in individual meetings, focus groups, or workshops in order to stimulate discussions and solicit feedback. Researchers proposed the idea to turn fairer disease benefits of vision videos for crowd or E by using them on social media platforms. In this talk, we want to consider the research question, what is the potential of using vision videos on social media platforms to solicit feedback in the form of video comments for crowd or E? We investigated this research question from three perspectives regarding the quantity of views and comments, the content of the comments, and their suitability for analysis used in crowd or E. In particular, we followed a clear research approach consisting of the four main steps, data collection, data cleaning, and manual analysis for collector data, and first, automated analysis. In the manual analysis, we spent a lot of effort in order to identify the different classification categories and create a VNT label data set, which can be used for the automated analysis. In particular, we selected YouTube as a platform and the vision video tunnels by the Warren Company. At the time, we downloaded the data set, the video had 4,500 comments and 2,100 replies. The comments are directly related to the video while the replies address some kind of discussions below the respective comment. For this reason, we focus only on this paper on the end of this subsequent comment. After the data cleaning, the data set contained 4,400 comments for the sub-C. and manual analysis. If you are interested in our data set scripts and the coding guides, the replication packets are provided on the node. Now let's consider some of the findings of the paper. For the perspective of quantity, how that after four days, there were already 2,770 comments. This is interesting, then these are 53% of all comments that were collected in four years. So we can say that a large number of comments was received in a really short amount of time. The result of the content perspective are based on a comprehensive manual analysis to collect the video comments. For this purpose, we had two coders who performed for us two classification tasks of the video comment regarding their relevance and polarity, as well as following up by the intentional topic. After each classification task, we had a discussion meeting to resolve any disagreements between the two coders. We found that a seven times 64 comments are only relevant from the point of view CROT or E. The further investigation of the intentional topics showed that the main intent is a feature request and problem report, which are addressed in the video comments. In addition, we found that the topics efficiency and safety are the most frequent topics addressed on the video comments. For the classification regarding the intentional topic, we used the taxonomy of user classification provided by Santa Cetari. What we found is that a lot of the content in the comments is related to the taxonomy of user feedback classification categories. Regarding the last perspective of suitability, we investigated how well the video comments can be classified by machine learning algorithms that are frequently applied in CROT or E. Our selection is based on the overview of user feedback classification approaches provided by Santa Cetari. We mainly use three algorithms and two feature. These algorithms support vector machines and they are based at the most frequently used ones in CROT or E. And in addition, we focus on the use of random forest which is rarely used but showed already good performance in different studies. For the features, we focused on back-of-words and TF-IDF. In particular, we created for each category a balanced stage set and performed a 10-fold cross validation for binary classification task. To reduce data splitting bias, we repeated each 10-fold cross-validation 10 times for each category. Our results show that SVM combined with TF-IDF achieved the best results overall. In particular, we found that the relevant comments and the topic of safety can be classified very well. However, the intention of problem report is the most difficult one to classify. Now we would like to discuss some of our findings. First of all, considering the feedback on safety. If we consider the taxonomy of user feedback classification, we see that it was developed with a systematic literature review on the topic of classifying user feedback. In particular, the authors investigated 43 different papers and identified 78 different categories for user feedback. However, safety is not an element of the taxonomy. And we assume that this lack of safety in the taxonomy is a result of the currently and frequently used user feedback sources, mainly user forums and mobile application markets. Vision videos are in particular concrete by showing the system and its use. From this way, they can empower stakeholders to experience and report their suspected quality news problems, such as economic health or safety risk, even if they have never actually used the system. This benefit is already known from applying videos in focus groups or workshops, for example. This observation and explanation let us conclude that we can see directly that one of the most important benefits of vision videos can be transferred on social media platforms, resulting in a currently completely new and even missing aspect of category in the taxonomy of user feedback classification. As a second aspect, we would like to discuss the selected vision media. We selected the vision media tunnels by the Boring Company, with company responsible by Elon Musk, and in this way, it already has a strong social media presence revolting in thousands of companies. However, this does not vote for average company. Therefore, we need to take strategy how we can build social media presence for companies in order to solicit thousands of video comments. Nevertheless, we need even more besides the social media presence, we also need to bridge the gap to the development. In this way, there's the need for holistic approaches for crowd-re, covering the entire software development process in order to operationally involve the crowd. As an answer to our research question, we can summarize that there's a potential of using vision videos on social media platforms. Regarding the number of commentant there are content with steep promising results for crowd-re. In this way, a vision videos can be suitable option to motivate in crowd numbers to actively participate by writing comments which are available source of feedback. Finally, I would like to conclude the presentation. What we've seen, we've considered the elicitation and the analysis of view of feedback in crowd-re. We've seen that frequently used sources are mobile application market in user forms. However, there are other sources and in particular social media platforms which are rarely used. We've considered the idea of using videos on social media platforms to solicit thousands of comments. We investigated the content of these comments and found interesting result which are promising regarding the use of vision videos for crowd-re. However, there's a lot of future work to do. First of all, there's new trend and text classification using deep learning algorithm. We would suggest to apply these algorithms also on our dataset to assess its usability for crowd-re. In addition, we need other replication and extensions of our study, including the development of first datasets. Due to the lack of suitable datasets, we spent a lot of effort in creating our labeled dataset which hopefully can be further used by other researchers. At the last point, we want to emphasize the need to investigate also the replies to cycle comments on the video. Based on our manual review, we saw that these applies do not address the video itself but show some kind of discussion below the associated comment. We would suggest to use, for example, frameworks like crowd-arrow proposed by Carnet R in order to identify arguments for or against a given statement. In this way, we could obtain further inflight into the source of the crowd. In this way, we want now to close our talk and thank you for your interest. Have a nice day.
video comments as a source of feedback. My name motivation opportunities for more than crowd meetings, focus groups, or workshops in order to stimulate discussions and solicit feedback. by using them on social media platforms. In this what is the potential of using vision videos on investigated this research question from three comments, the content of the comments, and their analysis. In particular, we selected YouTube as a this reason, we focus only on this paper on the 2,770 comments. This is interesting, then these are 53% of all comments that were collected in four years. So we can say that a large number of comments was received in a really short amount of a feature request and problem report, which are how well the video comments can be classified by machine learning algorithms that are frequently particular, we found that the relevant comments and the topic of safety can be classified very classifying user feedback. In particular, the application markets. Vision videos are in for crowd-re. In this way, a vision videos can be actively participate by writing comments which We investigated the content of these comments and regarding the use of vision videos for crowd-re. that these applies do not address the video