---
abstract: 'In this paper, we propose the task of live comment generation. Live comments are a new form of comments on videos, which can be regarded as a mixture of comments and chats. A high-quality live comment should be not only relevant to the video, but also interactive with other users. In this work, we first construct a new dataset for live comment generation. Then, we propose a novel end-to-end model to generate the human-like live comments by referring to the video and the other users’ comments. Finally, we evaluate our model on the constructed dataset. Experimental results show that our method can significantly outperform the baselines.[^1]'
author:
- |
    Damai Dai\
    School of EECS, Peking University\
    [daidamai@pku.edu.cn]{}
bibliography:
- 'emnlp2018.bib'
title: Live Video Comment Generation Based on Surrounding Frames and Live Comments
---

Introduction
============

In this paper, we focus on the task of automatically generating live comments. Live comments (also known as “

[UTF8]{}[gbsn]{}弹幕

bullet screen” in Chinese) are a new form of comments that appear in videos. We show an example of live comments in Figure \[fig:live commenting\]. Live comments are popular among youngsters as it plays the role of not only sharing opinions but also chatting. Automatically generating live comments can make the video more interesting and appealing.

Different from the other video-to-text tasks, such as video caption, a live comment appears at a certain point of the video timeline, which gives it some unique characteristics. The live comments can be causal chats about a topic with other users instead of serious descriptions of the videos. Therefore, a human-like comment should be not only relevant to the video, but also interactive with other users.

![An example of live comments. The colorful Chinese characters are the real-time live comments.[]{data-label="fig:live commenting"}](dm3.png)

![image](danmu.jpg)

In this paper, we aim at generating human-like live comments for the videos. We propose a novel end-to-end model to generate the comments by referring to the video and other users’ comments. We have access to not only the current frame but also the surrounding frames and live comments because a live comment and its associated frame are in the context of a series of surrounding frames and live comments. To make use of the information in those two parts, we design a model that encodes the surrounding frames and live comments together into a vector, based on which we decode the new live comment. Experimental results show that our model can generate human-like live comments.

Our contributions are two folds:

-   We propose a new task of automatically generating live comments and build a dataset with videos and live comments for live comment generation.

-   We propose a novel joint video and live comment model to make use of the current frame, the surrounding frames, and the surrounding live comments to generate a new live comment. Experimental results show that our model can generate human-like live comments.

Proposed Model {#model}
==============

In Figure \[fig:model\] we show the architecture of our model. Our live comment generation model is composed of four parts: a video encoder, a text encoder, a gated component, and a live comment generator. The video encoder encodes $n$ consecutive frames, and the text encoder encodes $m$ surrounding live comments into the vectors. The gated component aggregates the video and the comments into a joint representation. Finally, the live comment generator generates the target live comment.

Video Encoder
-------------

In our task, each generated live comment is attached with $n$ consecutive frames. In the video encoding part, each frame $f^{(i)}$ is first encoded into a vector $v_{f}^{(i)}$ by a convolution layer. We then use a GRU [@cho2014learning] layer to encode all the frame vectors into their hidden states $h_{v}^{(i)}$: $$\begin{aligned}
v_{f}^{(i)} &= CNN(f^{(i)}) \\
h_{v}^{(i)} =& GRU(v_{f}^{(i)}, h_{v}^{(i-1)})\end{aligned}$$ We set the last hidden state $h_{v}^{(n)}$ as the representation of the video $v_v = h_v^{(n)}$.

Text Encoder
------------

In the comment encoding part, a live comment $c^{(i)}$ with $L^{(i)}$ words $(w_{1}^{(i)},w_{2}^{(i)},\cdots,w_{L^{(i)}}^{(i)})$ is first encoded into a series of word-level hidden states ($h_{w_{1}}^{(i)},h_{w_{2}}^{(i)},\cdots,h_{w_{L^{(i)}}}^{(i)}$), using a word-level GRU layer. We use the last hidden state $h_{L^{(i)}}^{(i)}$ as the representation for $c^{(i)}$ denoted as $v_{c}^{(i)}$. Then we use a sentence-level GRU layer to encode all the live comment vectors of different live comments into their hidden states $h_{c}^{(i)}$: $$\begin{aligned}
h_{w_{j}}^{(i)} = GRU(w_{j}^{(i)}, h_{w_{j-1}}^{(i)})\\
h_c^{(i)} = GRU(v_{c}^{(i)}, h_c^{(i-1)})
$$ The last hidden state $h_{c}^{(m)}$ is used as the vector of all the live comments $v_c = h_c^{(m)}$.

Gated Selection
---------------

In order to describe how much information we should get from the video and the live comments, we apply a gated multi-layer perceptron (MLP) to combine $v_{c}$ and $v_{v}$, and get the final vector $h$: $$\begin{aligned}
s_{v} = u\mathop{ReLU}(W_{v}v_{v}+b_{v}) \\
s_{c} = u\mathop{ReLU}(W_{c}v_{c}+b_{c}) \\
h = 
\left[ 
    \begin{array}{cc}
        \frac{e^{s_c}}{e^{s_v}+e^{s_c}}v_c, & \frac{e^{s_v}}{e^{s_v}+e^{s_c}}v_v \\
    \end{array} 
\right ]\end{aligned}$$ where $u,W,b$ are trainable parameters.

Live Comment Generator
----------------------

We use a GRU to decode the live comment. The encoder encodes the frames and live comments jointly into a vector $h$. The probability of generating a sentence given the encoded vector $h$ is defined as, $$p(w_0,...,w_T|h) = \prod_{t=1}^{T}p(w_t|w_0,...,w_{t-1},h)$$ More specifically, the probability distribution of word $w_{i}$ is calculated as follows, $$\begin{aligned}
h_i =& GRU(w_{i}, h_{i-1})\\
p(w_{i}|w_0,...,w_{i-1},&h) = softmax(W_{o}h_i)\end{aligned}$$

Experiments
===========

In this section, we show the experimental results of our proposed model and compare it with three baselines on the dataset we construct from Youku.[^2]

Live Comment Dataset Construction
---------------------------------

**Video frames:** We extract frames from an animated TV series named “Tianxingjiuge”

[UTF8]{}[gbsn]{} (“天行九歌”)

at a frequency of 1 frame per second. We get frames from 40 videos in total, with a shape of $128 \times 72$ for each frame. We split the frames into training set () and test set (600).

**Live comments:** We use the developer tools in Google Chrome to manually detect the live comments sources, via which we get all live comments of the 40 videos. For each extracted frame, we select 5 live comments which are the nearest to the frame at the time they appear.

**Reference set:** Besides the live comments in our training set and test set, we crawl extra live comments to be the reference set for calculating BLEU score and perplexity which can evaluate the fluency of generated live comments (refer to Table \[bleu\_ppl\]).

**Copyright statement:** The dataset we construct can only be used for scientific research. The copyright belongs to the original website **Youku**.

Baselines
---------

Besides the model described in section  \[model\], we have three baseline methods:

**Frame-to-Comment (F2C)** [@vinyals2015show] applies a CNN to encode the current frame to a vector, based on which the decoder generates the target live comment.

**Moment-to-Comment (M2C)** applies an RNN to make use of one live comment near the current frame besides the CNN for the frame. The two encoded vectors are concatenated to be the initial hidden state for the decoder.

**Context-to-Comment (C2C)** is similar to  [@venugopalan2015sequence] which makes use of a series of surrounding frames and live comments by encoding them with extra higher-level RNNs.

Evaluation Metrics
------------------

We design two types of evaluation metrics: **human evaluation** and **automatic evaluation**.

**Human Evaluation:** We evaluate in three aspects: **Fluency** is designed to measure whether the generated live comments are fluent setting aside the relevance to videos. **Relevance** is designed to measure the relevance between the generated live comments and their associated frames. **Overall Score** is designed to synthetically measure the confidence that the generated live comments are made by humans in the context of the video. For all of the above three aspects, we stipulate the score to be an integer in $\{1, 2, 3, 4, 5\}$. The higher the better. The scores are evaluated by three seasoned native speakers and finally we take the average of three raters as the final result.

**Automatic Evaluation:** We adopt two metrics: **BLEU score** [@Papineni:2002:BMA:1073083.1073135] and **perplexity**. These two metrics are designed to measure whether the generated live comments accord with the human-like style. To get **BLEU scores**, for each generated live comment, we calculate its BLEU-4 score with all live comments in the reference set, and then we pick the maximal one to be its final score. **Perplexity** is to measure the language quality of the generated live comments, which is estimated as, $$perplexity = 2^{-\frac{1}{n}\sum_{i}\log{p(w_i \vert h_i)}}$$ for each word $w_i$ in the sentence, $h_i$ is the predicted word.

Experimental Details
--------------------

The vocabulary is limited to be the 34,100 most common words in the training dataset. We use a shared embedding between encoder and decoder and set the word embedding size to 300. The word embedding is randomly initialized and learned automatically during the training.

For the 3 GRUs used in the encoding stage, we set the hidden size to 300. For the decoding GRU, we set the hidden size to 600. For the encoding CNN, we use 3 convolution layers and 3 linear layers, and get a final vector with a size of 300. The batch size is 512. We use the Adam [@KingmaBa2014] optimization method to train the model. For the hyper-parameters of Adam optimizer, we set the learning rate $\alpha = 0.0003$, two momentum parameters $\beta_{1} = 0.9$ and $\beta_{2} = 0.999$ respectively, and $\epsilon = 1 \times 10^{-8}$. During training, we use “teacher forcing” [@Williams1989Experimental] to make our model converge faster and we set the teacher forcing ratio $p = 0.5$.

Results and Analysis
--------------------

As shown in Table \[manual\], our model achieves the highest scores over the baseline models in all three degrees. When only the current frame or one extra live comment is considered (**F2C** and **M2C**), the generated live comments have low scores. After considering more surrounding frames and live comments (**C2C**), all of the scores get higher. Finally, with the gate mechanism that can automatically decide the weights of surrounding frames and live comments, our **proposal** achieves the highest scores, which are almost near to those of real-world live comments. We use Spearman’s Rank correlation coefficients to evaluate the agreement among the raters. The coefficients between any two raters are all near 0.6 and at an average of 0.63. These high coefficients show that our human evaluation scores are consistent and credible. **Relevance:** The relevant scores presented in Table \[manual\] show that the live comments generated by all models do not achieve high relevant scores, which means that many of the generated live comments are not relevant to the current frame. We go through the real live comments and find that about 75.7% live comments are not relevant to the current frame, but are just chatting. In fact, we can find from Table \[manual\] that the relevance score of real live comments is not high as well. Therefore, the low relevant scores are reasonable. Still, our proposal can generate more relevant live comments owing to its ability to combine the information from the surrounding frames and live comments.

**Fluency:** From the fluency score presented in Table \[manual\], the BLEU-4 score and the perplexity score presented in Table \[bleu\_ppl\], we can see our proposal can generate live comments which best accord with the human-like style. **Informativeness:** From the *Average Length* in Table \[bleu\_ppl\], we can see our proposal improves the length of the generated live comments, which indicates that more meaningful information is embodied.

Related Work
============

Inspired by the great success achieved by the sequence-to-sequence learning framework in machine translation [@sutskever2014sequence; @cho2014learning; @bahdanau2014neural], @vinyals2015show and @mao2014explain proposed to use a deep convolutional neural network (CNN) to encode the image and a recurrent neural network (RNN) to generate the image captions. @xu2015show further proposed to apply attention mechanism to focus on certain parts of the image when decoding. Using CNN to encode the image while using RNN to decode the description is natural and effective when generating textual descriptions.

One task that is similar to live comment generation is image caption generation, which is an area that has been studied for a long time. @farhadi2010every tried to generate descriptions of an image by retrieving from a big sentence pool. @kulkarni2011baby proposed to generate descriptions based on the parsing result of the image with a simple language model. These systems are often applied in a pipeline fashion, and the generated description is not creative. More recent work is to use stepwise merging network to improve the performance [@liu2018].

Another task which is similar to this work is video caption generation. @venugopalan2015sequence proposed to use CNN to extract image features, and use LSTM to encode them and decode a sentence. Similar models [@shetty2016frame; @jin2016describing; @ramanishka2016multimodal; @dong2016early; @pasunuru2017multi; @shen2017weakly] are also proposed to handle the task of video caption generation. @Das2017ICCV [@Das2017CVPR] introduce the task of Visual Dialog, which requires an AI agent to answer a question about an image when given the image and a dialog history.

We cast this problem as a natural language generation problem, and we are also inspired by the recent related work of natural language generation models with the text inputs [@DBLP:journals/corr/abs-1803-01465; @DBLP:journals/corr/abs-1805-04871; @DBLP:journals/corr/abs-1805-05181; @DBLP:journals/corr/abs-1802-01345].

Conclusion
==========

In this paper, we propose the task of live comment generation. In order to generate high-quality comments, we propose a novel neural model which makes use of the surrounding frames in the video and other surrounding live comments. Experimental results show that our model performs better than the baselines in various metrics, and even approaches the performance of human.

[^1]: The dataset and the code will be released to the public if the manuscript is accepted.

[^2]: <http://www.youku.com>
