\subsection{Personal unsafety}
\vspace{-1mm}
Personal unsafe responses have a direct negative impact on users, such as causing psychological or physical harm by attacking, mocking, etc., or inducing potentially risky behaviors by spreading unreliable information.
Specifically, we focus on the following three categories.

\noindent \textbf{Offending User} \quad
The generated responses should not be aggressive or offensive, satire intended to ridicule or insult \cite{dinan2019build}, and any other statements intended to enrage user \cite{sheng2021nice}. Offensiveness based on context can be more implicit and even more infuriating (e.g. cursing back, evil for good, etc.).

\noindent \textbf{Risk Ignorance} \quad
Previous studies pay much attention to mental health risks potentially carried by the outputs of generative model \cite{abd2020effectiveness, sun-etal-2021-psyqa}. It is notable that mental health risk may also induce physical health dangers (e.g. suicide).
We warn risk ignorance, which may distress the users or even cause irreparable injury.

\noindent \textbf{Unauthorized Expertise} \quad
For general chatbots, it is unsafe to provide plausible suggestions, counsels, and knowledge without professional qualifications, especially in safety-critical fields like medical and legal domains \cite{mittal2016comparative}.
Here we primarily focus on unauthorized medical expertise.

\vspace{-2mm}
\subsection{Non-personal Unsafety}
Non-personal unsafe responses are primarily toward third-party individuals, groups, and the social mass.
We focus on three categories as follows.

\noindent \textbf{Toxicity Agreement}  \quad
Previous work finds that chatbots tend to show agreement or acknowledgment faced with toxic context \cite{Baheti2021JustSN}.
Such responses advocate users’ harmful speech, spread toxicity, rude or bias in an indirect form \cite{dinan2021anticipating}.

\noindent \textbf{Biased Opinion}  \quad
Biased opinion usually maintains stereotypes and prejudices, referring to negative expressions on individuals or groups based on their social identities (e.g., gender and race) \cite{Blodgett2020BiasSurvey}.
In this paper, we primarily focus on biased opinions on gender, race, and religion.

\noindent \textbf{Sensitive Topic Continuation} \quad
Some topics are more controversial than others, and showing disposition or preference in one way can potentially upset some certain groups of users \cite{xu2020recipes}.
We regard responses continuing the same sensitive topics of the context and expressing views or preferences as unsafe cases.

\subsection{Data Source}
We collect data from the following three sources.

\noindent \textbf{Real-world Conversations} \quad
The majority of our data are real-world conversations from Reddit because of their better quality, more varieties, and higher relevance than model generated samples. We collect post-response pairs from Reddit
by PushShift API \cite{Baumgartner2020ThePR}. We create a list of sub-reddits for each category of context-sensitive unsafety, where it is easier to discover unsafe data. Refer to Appendix \ref{apx:real-col} for the details of real-world conversations collection.

\noindent \textbf{Public Datasets} \quad
We notice that some existing public datasets can be modified and used under the definition of certain categories of our proposed taxonomy.  Therefore, we add them to our dataset candidates. For instance, MedDialog \cite{Zeng2020MedDialog} are composed of single-turn medical consulting. However, it is not appropriate for general conversational models to give such professional advice like that. Thus we add MedDialog dataset as our unsafe data candidates in \textit{Unauthorized Expertise}. Also, \citet{sharma2020empathy} releases some contexts related to mental health and corresponding empathetic responses from Reddit, which we regarded as safe data candidates in \textit{Risk Ignorance}.

\noindent \textbf{Machine-generated Data} \quad
It is naturally beneficial to exploit machine-generated data to research on the safety of neural conversational models themselves.
We take out the prompt/context of our collected data including real-world conversations and public dataset and let conversational models generate responses. According to the characteristics of each unsafe category, we try to find prompts that are more likely to induce unsafety. Refer to Appendix \ref{apx:gen-data} for detailed prompting picking methods and generating based on prompting.

After collecting from multiple sources, we do a post-processing for data cleaning including format regularization and explicit utterance-level unsafety filtering (refer to Appendix \ref{apx:post}).

\vspace{-1mm}
\subsection{Human Annotation}

\paragraph{Semi-automatic Labeling}

It is helpful to employ auto labeling method to improve annotation efficiency by increasing the recall of context-sensitive unsafe samples. For some certain unsafe categories, we find there are some patterns that classifiers can find to separate the safe and unsafe data according to the definitions. For \textit{Unauthorized Expertise}, we train a classifier to identify phrases that offer advice or suggestions for medicine or medical treatments. For \textit{Toxicity Agreement}, we train a classifier to identify the dialogue act ``showing agreement or acknowledgement'' based on the SwDA dataset \cite{Jurafsky-etal:1997} and manually picked data. To verify the auto-labeling quality, we randomly pick 200 samples and do human confirmation in Amazon Mechanical Turk (AMT) platform (\url{mturk.com}) as the golden labels. We compute the accuracy shown in Table \ref{tab:dataset_stat} and all are higher than 92\%, which proves that our auto labeling method is valid.

For \textit{Risk Ignorance}, \textit{Offending User}, and \textit{Biased Opinion}, there are few easy patterns to distinguish between the safe and unsafe data. Thus the collected data from the three unsafe categories are completely human-annotated. For each unsafe category, we release a separate annotation task on AMT and ask the workers to label safe or unsafe. Each HIT is assigned to three workers and the option chosen by at least two workers is seen as the golden label. We break down the definition of safety for each unsafe category, to make the question more intuitive and clear to the annotator. Refer to Appendix \ref{apx:guideline} for the annotation guidelines and interface. We do both utterance-level and context-level annotations to confirm that the final dataset is context-sensitive.

\vspace{-2mm}
\paragraph{Utterance-level Annotation}
We take another round of human annotation to ensure that all of our responses are utterance-level safe, though post-processing filters out most of the explicitly unsafe samples. For each context-response pair, only the response is provided to the annotator who is asked to label whether the response is unsafe.
\vspace{-2mm}
\paragraph{Context-level Annotation}
For those data which is safe in utterance-level annotation, we conduct context-level annotation, where we give both the context and the response to the annotators and ask them whether the response is safe given the conversational context. If the data is safe, we add them into the safe part of our dataset, vice versa.

\vspace{-2mm}
\paragraph{Model-in-the-loop Collection}
To improve collection efficiency, our data collection follows a model-in-the-loop setup. We train a classifier to discover context-sensitive unsafe responses from the ocean of responses. We pick the data samples with comparatively high unsafe probability and send them to be manually annotated by AMT workers. Annotation results in return help train the classifier to get better performance to discover context-sensitive unsafe responses. We initialize the classifier by labeling 100 samples ourselves and we repeat the process above three times.

\vspace{-1mm}
\subsection{Annotation Quality Control}
Only those workers who arrive at 1,000 HITs approved and 98\% HIT approval rate can take part in our tasks. Besides, we limit workers to native English speakers by setting the criterion ``location''. The workers are aided by detailed guidelines and examples (refer to Appendix \ref{apx:guideline}) during the annotation process. We also embed easy test questions into the annotations and reject HITs that fail the test question. The remuneration is set to approximately 25 USD per hour. We gradually enhance our annotation agreement by improving and clarifying our guidelines. As shown in Table \ref{tab:dataset_stat}, the overall annotations achieve moderate inter-annotator agreement.\footnote{Comparable to the related contextual tasks which gets krippendorff's alpha $\alpha=0.22$~\cite{Baheti2021JustSN}.}

\subsection{Experimental Setup}
To answer first two questions, we first construct a unsafety\footnote{In this section, we use ``unsafety'' to refer to ``context-sensitive unsafety'' for convenience.} detector.
We randomly split our dataset into train (80\%), dev (10\%), and
test (10\%) sets for each category of unsafety. And we use RoBERTa model \cite{ roberta2019} with 12 layers for our experiments, which has shown strong power in text classification tasks. We input the context and response with \texttt{</s>} as the separator.

We construct five one-vs-all classifiers, one for each unsafe category, and combines the results of five models to make the final prediction.
That is, each model performs a three-way classification (Safe, Unsafe, N/A) for one corresponding unsafe category.
In real-world tests, the coming data may belong to other unsafe categories. To prevent the models from failing to handle the unknown unsafe categories, we add a ``N/A'' (Not Applicable) class and its training data is from other categories (both safe and unsafe), expecting the models to identify data out of domain. We classify a response as: (1) \textbf{Safe} if all five models determine the response is safe or N/A; (2) \textbf{Unsafe in category $\mathbf{C}$} if the model for $\mathbf{C}$ determines the response is unsafe. If multiple models do so, we only consider the model with the highest confidence. We compare this method with a single model which trains on mixed data in one step, which is detailed in Appendix \ref{apx:exp}.

\begin{table}[tbp]
\centering
\scalebox{0.80}{
\begin{tabular}{@{}c|rrr|rrr@{}}
\toprule
\multicolumn{1}{l|}{\multirow{2}{*}{\textbf{Class}}} & \multicolumn{3}{c|}{\textbf{With Context (\%)}} & \multicolumn{3}{c}{\textbf{W/o Context (\%)}} \\
\multicolumn{1}{l|}{} & \multicolumn{1}{c}{Prec.} & \multicolumn{1}{c}{Rec.} & \multicolumn{1}{c|}{F1} & \multicolumn{1}{c}{Prec.} & \multicolumn{1}{c}{Rec.} & \multicolumn{1}{c}{F1} \\ \midrule
\textbf{Safe} & 87.8 & 85.9 & 86.8 & 82.4 & 80.0 & 81.2 \\
\textbf{OU} & 82.5 & 88.0 & 85.2 & 53.8 & 76.0 & 63.0 \\
\textbf{RI} & 78.9 & 75.5 & 77.2 & 62.4 & 56.4 & 59.2 \\
\textbf{UE} & 96.6 & 92.5 & 94.5 & 90.4 & 91.4 & 90.9 \\
\textbf{TA} & 94.5 & 94.5 & 94.5 & 76.7 & 85.6 & 80.9 \\
\textbf{BO} & 61.4 & 71.4 & 66.0 & 56.0 & 42.9 & 48.6 \\
\midrule
\textbf{Overall} & \textbf{83.6} & \textbf{84.6} & \textbf{84.0} & 70.3 & 72.0 & 70.6 \\ \bottomrule
\end{tabular}}
\caption{Results of fine-grain classification by one-vs-all classifiers between with and without context.}
\label{tab:7cls-res}
\vspace{-3mm}
\end{table}

\subsection{Fine-grain Classification}
\label{sec:fine-cls}
Given a pair of context and response, the fine-grain classification task requires models to identify whether a response is unsafe and then which unsafe category the response belongs to. We classify according to the rule above and Table \ref{tab:7cls-res} shows the experimental results.

The comparatively high performance shows that the neural models can effectively discover the implicit connections between context and response, then identify context-sensitive unsafety. Meanwhile, we notice the model gets a relatively low F1-score in \textit{Biased Opinion}. We believe that in this category, the complexity and sample-sparsity of the social identities
(e.g. LGBT, Buddhist, blacks, etc.)
are huge obstacles for a neural model without external knowledge to learn.

Besides, for exploring how much influence context has on context-sensitive unsafety detection, we do an ablation study and compare the classifier performance between with context and without context. As shown in Table \ref{tab:7cls-res}, The absolute improvement of the overall F1 score is high to 13.4\%. It verifies that in our dataset, the context is indeed the key information to determine whether the response is safe or not. Also, we notice that by adding context, \textit{Unauthorized Expertise} improve less obviously, which accords with our expectation. UE is seen context-sensitive unsafe due to the context of human-bot dialogue setting, while the detection itself may be quite easy at utterance-level like matching medicine and suggestion-related words in response.
We also conduct the same experiments as above by constructing a single classifier (refer to Appendix \ref{apx:exp}). It shows that one-vs-all classifiers perform slightly better in all categories.

\begin{table}[tbp]
\centering
\scalebox{0.72}{
\begin{tabular}{lcrrrrr}
\toprule
\multirow{2}{*}{Methods}  & \multicolumn{1}{c|}{\multirow{2}{*}{Inputs}} & \multicolumn{1}{r|}{Safe}    & \multicolumn{1}{r|}{Unsafe}  & \multicolumn{3}{c}{Macro Overall (\%)}                                        \\
                          & \multicolumn{1}{c|}{}                        & \multicolumn{1}{c|}{F1 (\%)} & \multicolumn{1}{c|}{F1 (\%)} & \multicolumn{1}{c}{Prec.} & \multicolumn{1}{c}{Rec.} & \multicolumn{1}{c}{F1} \\ \midrule
Random                    & \multicolumn{1}{c|}{N/A}                     & \multicolumn{1}{r|}{53.5}    & \multicolumn{1}{r|}{48.1}    & 50.9                      & 50.9                     & 50.8                   \\ \midrule
\multirow{2}{*}{Detoxify} & \multicolumn{1}{c|}{Resp}                    & \multicolumn{1}{r|}{70.4}    & \multicolumn{1}{r|}{9.9}     & 60.5                      & 51.5                     & 40.1                   \\
                          & \multicolumn{1}{c|}{(Ctx,resp)}              & \multicolumn{1}{r|}{61.7}    & \multicolumn{1}{r|}{56.9}    & 59.3                      & 59.4                     & 59.3                   \\ \midrule
\multirow{2}{*}{P-API}    & \multicolumn{1}{c|}{Resp}                    & \multicolumn{1}{r|}{70.2}    & \multicolumn{1}{r|}{11.5}    & 58.3                      & 51.5                     & 40.8                   \\
                          & \multicolumn{1}{c|}{(Ctx,resp)}              & \multicolumn{1}{r|}{58.8}    & \multicolumn{1}{r|}{57.7}    & 58.5                      & 58.6                     & 58.3                   \\ \midrule
 BBF & \multicolumn{1}{c|}{(Ctx,resp)} & \multicolumn{1}{r|}{62.8} & \multicolumn{1}{r|}{55.9} & 59.3 & 59.3 & 59.3 \\ \midrule
 BAD & \multicolumn{1}{c|}{(Ctx,resp)} & \multicolumn{1}{r|}{71.1} & \multicolumn{1}{r|}{61.8} & 66.9 & 66.4 & 66.5 \\ \bottomrule
\multicolumn{7}{c}{After finetuning on \textsc{DiaSafety}}                                                                                \\ \toprule
Detoxify                  & \multicolumn{1}{c|}{(Ctx,resp)}              & \multicolumn{1}{r|}{80.8}    & \multicolumn{1}{r|}{79.0}    & 79.9                      & 80.1                     & 79.9                   \\ \midrule
Ours                      & \multicolumn{1}{c|}{(Ctx,resp)}              & \multicolumn{1}{r|}{86.8}    & \multicolumn{1}{r|}{84.7}    & 85.7                      & 85.8                     & 85.7                   \\ \bottomrule
\end{tabular}}
\caption{Coarse-grain classification results on our test set using different methods. PerspectiveAPI and Detoxify without finetuning on \textsc{DiaSafety} only accept single utterance. Thus we test by (1) inputting only response and (2) concatenating context and response to make them access to the information of context. We report the complete results in Appendix \ref{apx:exp2}.}
\label{tab:2cls-res}
\vspace{-4mm}
\end{table}

\subsection{Coarse-grain Classification}
\label{sec:exp-cc}
To check whether existing safety guarding tools can identify our context-sensitive unsafe data, we define a coarse-grain classification task, which merely requires models to determine whether a response is safe or unsafe given context.

\noindent \textbf{Deceiving Existing Detectors} \quad
PerspectiveAPI (\textbf{P-API}, \url{perspectiveapi.com}) is a free and popular toxicity detection API, which is used to help mitigate toxicity and ensure healthy dialogue online. \textbf{Detoxify}~\cite{Detoxify} is an open-source RoBERTa-based model trained on large-scale toxic and biased corpora. Other than utterance-level detectors, we also test two context-aware dialogue safety models: Build it Break it Fix it (\textbf{BBF}) \cite{dinan2019build} and Bot-Adversarial Dialogue Safety Classifier (\textbf{BAD}) \cite{xu-etal-2021-bot}. We check these methods on our test set and add a baseline that randomly labels safe or unsafe.
As shown in Table \ref{tab:2cls-res},
Detoxify and P-API get a quite low F1-score (close to random no matter what inputs). When inputs contain only response, the recall of unsafe responses is especially low, which demonstrates again that our dataset is context-sensitive.
Meanwhile, we notice that both methods get a considerable improvement by adding context. We attribute that to the fact that contexts in some unsafe samples carrying toxic and biased contents (e.g. \textit{Toxicity Agreement}). Besides, Our experimental results demonstrate that the context-aware models are still not sensitive enough to the context.
We consider that in the context-aware cases, a large number of unsafe responses which could be detected at the utterance level as a shortcut, make context-aware models tend to ignore the contextual information and thus undermine their performances.
In summary, our context-sensitive unsafe data can easily deceive existing unsafety detection methods, revealing potential risks.

\noindent \textbf{Improvement by Finetuning} \quad
We test the performance of Detoxify finetuned on \textsc{DiaSafety} (shown in Table \ref{tab:2cls-res}).
The experimental results show that Detoxify gets a significant improvement after finetuning.
Besides, we compare it with our coarse-grain classifier according to the rule that a response is determined to be unsafe if any one of the five models determines unsafe, otherwise the response is safe.
The main difference lies in that our classifier is finetuned from a vanilla RoBERTa, while Detoxify is pre-trained on an utterance-level toxic and biased corpus before finetuning.
Noticeably, we find pre-training on utterance-level unsafety detection degrades the performance to detect context-sensitive unsafety due to the gap in data distribution and task definition.
The results suggest that splitting the procedure of detecting utterance-level and context-sensitive unsafety is a better choice to perform a comprehensive safety evaluation.

\subsection{Two-step Safety Detection Strategy}
 Recall that dialogue safety of conversational models includes utterance-level and context-sensitive safety. As Section \ref{sec:exp-cc} shows, checking them separately not only seamlessly fuses utterance-level research resources with the context-sensitive dialogue safety task, but is also more effective.

Given a pair of context and response, in the first step, we employ Detoxify
and check whether the response is utterance-level unsafe; in the second step where the response passes utterance-level check, we utilize our classifiers to check whether the response becomes unsafe with adding context. This method, taking full advantage of the rich resources in utterance-level research, comprehensively checks the safety of conversational models.\footnote{Detoxify gets 93.7\% AUC score in its test set and ours get 84.0\% F1 score as above, which is reliable to some degree.}

\begin{figure*}[tbp]
  \centering
  \includegraphics[width=1.0\linewidth]{figures/compose4.pdf}
  \caption{Evaluation results triggered by 5 categories of contexts among different conversational models. We label the context-sensitive unsafe proportion (smaller score) and total unsafe proportion (larger score) for each bar. ``Overall'' is computed by macro average of five unsafe categories.}
  \label{fig:eval_res}
\vspace{-3mm}
\end{figure*}

\vspace{-1mm}
\subsection{Unsafety Metric}
We calculate scores regarding 5 categories of context-sensitive unsafety and utterance-level unsafety.
For a category $\mathbf{C}$, we take out the contexts of validation and test set in $\mathbf{C}$ as adversarial examples (also including those safe data). The evaluated model $\mathbf{M}$ generates 10 responses for each context.
Context in $\mathbf{C}$ may trigger (a) context-sensitive unsafe responses in $\mathbf{C}$
and (b) utterance-level unsafe responses.
We calculate the proportions of (a) and (b) to all responses in category $\mathbf{C}$. The lower the proportion is, the safer the model is.

\vspace{-1mm}
\subsection{Evaluated Models}
We evaluate three open-source conversational models which are publicly available. \textbf{DialoGPT} \cite{zhang2020dialogpt} extends GPT-2 \cite{radford2019language}  by fintuning on Reddit comment chains. \textbf{Blenderbot} \cite{roller2020blenderbot} is finetuned on multiple dialogue corpora \cite{smith-etal-2020-put} to blender skills. Moreover, Blenderbot is supposed to be safer by rigorously cleaning training data and augmenting safe responses \cite{xu2020recipes}. \textbf{Plato-2} \cite{bao2021plato2} introduces curriculum learning and latent variables to form a better response.

\vspace{-1mm}
\subsection{Evaluation Results}
\noindent \textbf{Among Different Models} \quad
As shown in Figure \ref{fig:eval_res}, Blenderbot has the best overall safety performance and the lowest unsafe proportion except for \textit{Toxicity Agreement}. We find Blenderbot tends to show agreement and acknowledgment to toxic context, which may be due to the goal of expressing empathy in training Blenderbot. Besides, Plato-2 is found weakest to control utterance-level safety. On the whole, existing conversational models are still stuck in safety problems, especially in context-sensitive safety. We sincerely call for future research to pay special attention on the context-sensitive safety of dialogues systems.

\noindent \textbf{Among Different Parameter Scales} \quad
Large conversational models have shown their superior in fluency, coherence and logical reasoning \cite{roller2020blenderbot, adiwardana2020meena}. However, from our experimental results shown in Figure \ref{fig:eval_res}, larger models do not come with safer responses. We analyze and speculate that larger models are over-confident in the aspect of unauthorized suggestions and implicit offensiveness while the smaller models are more cautious about the outputs and tend to generate general responses. In addition to Blenderbot, we extend our evaluation to more parameter scales of DialoGPT and Plato-2 and present a dialogue safety leaderboard which ranks 8 models in total in Appendix \ref{apx:eval}.

\noindent \textbf{Among Different Sampling Methods} \quad
Decoding algorithms have an important impact on the generation. We evaluate different sampling methods including top-$k$ sampling and nucleus sampling~\cite{holtzman2019curious} on DialoGPT and Blenderbot (shown in Appendix \ref{apx:eval}). We conclude that sampling methods have little impact on the safety of conversational models.

\section{Data Collection Details}
\subsection{Real-world Conversations}
\label{apx:real-col}
Context-sensitive unsafe data is rare in the Reddit corpus, especially after many toxic or heavily down-voted posts were already removed by moderators. Thus we adopt the following strategies to improve collection efficiency. (1) Keyword query. We query from the entire PushShift Reddit corpus for relevant keywords, and then extract the identified post and all its replies; for example, we search the keywords \textit{Asian people} to look for biased conversation pairs against this racial group. (2) Removing generally safe subreddits. There are many popular subreddits that are considered to be casual and supportive communities including r/Music, r/food, r/animations, etc. We remove posts from those communities to increase unsafe probability.

\subsection{Machine-generated Data}
\label{apx:gen-data}
 Prompts for generation have two major sources, (1) crawled using keyword query from Reddit, for \textit{Biased Opinion} dataset (2) collected from existing toxicity datasets, including the ICWSM 2019 Challenge \cite{mathew2018thou} and Kaggle Toxic Comment Classification Challenge\footnote{\url{https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/data}} for \textit{Toxicity Agreement} dataset. For \textit{Unauthorized Expertise}, we collect some utterances from MedDialog dataset \cite{Zeng2020MedDialog}. For \textit{Risk Ignorance}, we collect some posts related to mental health from epitome \cite{sharma2020empathy} and dreaddit \cite{turcan-mckeown-2019-dreaddit}. Given the collected prompts, We then generate responses using DialoGPT \cite{zhang2020dialogpt} and Blenderbot \cite{roller2020blenderbot} to construct context-response pair candidates.

\subsection{Post-processing}
\label{apx:post}
In data post-processing, we only retain context and response of length less than 150 tokens, and remove emojis, URLs, unusual symbols, and extra white spaces. Since our unsafe data is expected to be context-sensitive, an additional processing step is to remove explicitly unsafe data that can be directly identified by utterance-level detectors. We use Detoxify \cite{Detoxify} to filter out replies with toxicity score over 0.3.

\section{Annotation Guidelines}
\label{apx:guideline}
We present the annotation interface in Figure \ref{fig:guidline-web} and summarize our guidelines in Figure \ref{fig:guidline}.

\section{Additional Classification Experiments}
\subsection{Fine-grain Classification}
\label{apx:exp}
The classifier can be constructed by (a) A single multi-class
classifier, which mixes data from all categories (safe + five unsafe categories) and trains a classifier in one step; (b) One-vs-all multi-class classification, which trains multiple models, one for each unsafe category, and combines the results of five models to make the final prediction.
Intuitively, the topic and style of contexts vary a lot in different categories. As an example, in \textit{Risk Ignorance}, the topic is often related to mental health (such as depression, self-harm tendency), which is rare in other categories.
Chances are that a single classification model exploits exceedingly the style and topic information, which is not desirable.
We do the same experiments for fine-grain classification as in Section \ref{sec:fine-cls} with single model. Table \ref{tab:7cls-res-single} shows the experimental results with context and without context.

\subsection{Coarse-grain Classification}
\label{apx:exp2}
We report the complete coarse-grain classification results shown in Table \ref{tab:2cls-res-apx}.
\begin{table*}[tbp]
\centering
\scalebox{1.0}{
\begin{tabular}{@{}lcrrrrrrrrr@{}}
\toprule
\multirow{2}{*}{Methods} & \multicolumn{1}{c|}{\multirow{2}{*}{Inputs}} & \multicolumn{3}{c|}{Safe (\%)} & \multicolumn{3}{c|}{Unsafe (\%)} & \multicolumn{3}{c}{Macro Overall (\%)} \\
 & \multicolumn{1}{c|}{} & \multicolumn{1}{c}{Prec.} & \multicolumn{1}{c}{Rec.} & \multicolumn{1}{c|}{F1} & \multicolumn{1}{c}{Prec.} & \multicolumn{1}{c}{Rec.} & \multicolumn{1}{c|}{F1} & \multicolumn{1}{c}{Prec.} & \multicolumn{1}{c}{Rec.} & \multicolumn{1}{c}{F1} \\ \midrule
Random & \multicolumn{1}{c|}{N/A} & 55.1 & 51.9 & \multicolumn{1}{r|}{53.5} & 46.6 & 49.8 & \multicolumn{1}{r|}{48.1} & 50.9 & 50.9 & 50.8 \\ \midrule
\multirow{2}{*}{Detoxify}
 & \multicolumn{1}{c|}{Resp} & 55.1 & 97.7 & \multicolumn{1}{r|}{70.4} & 65.9 & 5.3 & \multicolumn{1}{r|}{9.9} & 60.5 & 51.5 & 40.1 \\
 & \multicolumn{1}{c|}{(Ctx,resp)} & 63.3 & 60.2 & \multicolumn{1}{r|}{61.7} & 55.3 & 58.5 & \multicolumn{1}{r|}{56.9} & 59.3 & 59.4 & 59.3 \\ \midrule
\multirow{2}{*}{PerspectiveAPI}
 & \multicolumn{1}{c|}{Resp} & 55.1 & 96.7 & \multicolumn{1}{r|}{70.2} & 61.5 & 6.3 & \multicolumn{1}{r|}{11.5} & 58.3 & 51.5 & 40.8 \\
 & \multicolumn{1}{c|}{(Ctx,resp)} & 63.3 & 54.9 & \multicolumn{1}{r|}{58.8} & 53.8 & 62.3 & \multicolumn{1}{r|}{57.7} & 58.5 & 58.6 & 58.3 \\ \midrule
 BBF & \multicolumn{1}{c|}{(Ctx,resp)} & 62.8 & 62.7 &\multicolumn{1}{r|}{62.8} & 55.8 & 55.9 & \multicolumn{1}{r|}{55.9} & 59.3 & 59.3 & 59.3 \\ \midrule
 BAD & \multicolumn{1}{c|}{(Ctx,resp)} & 68.0 & 74.5 & \multicolumn{1}{r|}{71.1} & 65.9 & 58.3 & \multicolumn{1}{r|}{61.8} & 66.9 & 66.4 & 66.5 \\
 BAD+Medical & \multicolumn{1}{c|}{(Ctx,resp)} & 70.9 & 50.6 & \multicolumn{1}{r|}{59.0} & 56.2 & 75.3 & \multicolumn{1}{r|}{64.4} & 63.5 & 62.9 & 61.7 \\
 \bottomrule

\multicolumn{11}{c}{After finetuning on \textsc{DiaSafety}} \\ \toprule
Detoxify & \multicolumn{1}{c|}{(Ctx,resp)} & 84.0 & 77.9 & \multicolumn{1}{r|}{80.8} & 75.8 & 82.4 & \multicolumn{1}{r|}{79.0} & 79.9 & 80.1 & 79.9 \\ \midrule
Ours & \multicolumn{1}{c|}{(Ctx,resp)} & 87.8 & 85.9 & \multicolumn{1}{r|}{86.8} & 83.6 & 85.8 & \multicolumn{1}{r|}{84.7} & 85.7 & 85.8 & 85.7 \\ \bottomrule
\end{tabular}}
\caption{Complete coarse-grain classification results on our test set using different methods. PerspectiveAPI and Detoxify without finetuning on \textsc{DiaSafety} only accept single utterance. Thus we test by (1) inputting only response and (2) concatenating context and response to make them access to the information of context. \citet{xu2020recipes} also present another medical topic classifier other than BAD classifier. We test responses in \textit{Unauthorized Expertise} using their medical topic classifier and use BAD classifier for other categories (shown in the row ``BAD+medical''). We find the result becomes even worse
because medical topic classifier recognizes topics but does not determine safe or not. Safe responses like ``maybe you should see a doctor'' are thus mislabeled.
}
\label{tab:2cls-res-apx}
\end{table*}

\begin{table}[tbp]
\scalebox{0.9}{
\begin{tabular}{@{}c|rrr|rrr@{}}
\toprule
\multicolumn{1}{l|}{\multirow{2}{*}{Category}} & \multicolumn{3}{c|}{With Context (\%)} & \multicolumn{3}{c}{W/o Context (\%)} \\
\multicolumn{1}{l|}{} & \multicolumn{1}{c}{Prec.} & \multicolumn{1}{c}{Rec.} & \multicolumn{1}{c|}{F1} & \multicolumn{1}{c}{Prec.} & \multicolumn{1}{c}{Rec.} & \multicolumn{1}{c}{F1} \\ \midrule
Safe & 88.9 & 80.0 & 84.2 & 86.4 & 74.7 & 80.1 \\
OU & 77.1 & 72.0 & 74.5 & 50.9 & 76.0 & 60.8 \\
RI & 66.1 & 87.2 & 75.2 & 55.8 & 51.1 & 53.3 \\
UE & 90.5 & 92.5 & 91.5 & 86.4 & 95.7 & 90.8 \\
TA & 91.3 & 93.8 & 92.6 & 67.9 & 85.6 & 75.8 \\
BO & 59.1 & 76.5 & 66.7 & 49.0 & 51.0 & 50.0 \\
\midrule
\textbf{Overall} & \textbf{78.9} & \textbf{83.7} & \textbf{80.8} & 66.1 & 72.4 & 68.5 \\ \bottomrule
\end{tabular}}
\caption{Results of our fine-grain classification by single model between with and without context. The unsafe categories are denoted by initials.}
\label{tab:7cls-res-single}
\end{table}

\section{Additional Evaluation Results}
\label{apx:eval}
We evaluate the safety of DialoGPT-Medium and Blenderbot-400M among different decoding parameters, which is shown in Figure \ref{fig:deco_res}.

Besides, as shown in Table \ref{tab:leaderboard}, we present a safety leaderboard of all of our evaluated models. In the leaderboard, we list utterance-level unsafe proportion as another column to more intuitively compare the performance of utterance-level safety.

\begin{table*}[tbp]
\centering
\begin{tabular}{@{}clrrrrrrc@{}}
\toprule
\textbf{Rank} & \multicolumn{1}{c}{\textbf{Models}} & \multicolumn{1}{c}{\textbf{OU}} & \multicolumn{1}{c}{\textbf{RI}} & \multicolumn{1}{c}{\textbf{UE}} & \multicolumn{1}{c}{\textbf{TA}} & \multicolumn{1}{c}{\textbf{BO}} & \multicolumn{1}{c}{\textbf{Utter}} & \multicolumn{1}{c}{\textbf{Overall}} \\ \midrule
\textbf{1}&\textbf{Blenderbot-S}&5.9&10.2&17.3&26.0&13.4&9.3&13.7\\
\textbf{2}&\textbf{Blenderbot-M}&4.5&9.2&14.7&45.0&5.4&3.7&13.7\\
\textbf{3}&\textbf{Blenderbot-L}&9.0&7.2&18.8&32.3&11.1&9.4&14.6\\
\textbf{4}&\textbf{Plato2-Base}&8.6&19.4&35.3&8.7&17.8&18.2&18.0\\
\textbf{5}&\textbf{Plato2-Large}&9.2&10.9&45.7&14.8&18.4&18.3&19.5\\
\textbf{6}&\textbf{DialoGPT-S}&17.4&45.1&27.8&16.6&28.3&7.5&23.8\\
\textbf{7}&\textbf{DialoGPT-M}&18.2&43.9&32.6&32.0&34.2&6.5&27.9\\
\textbf{8}&\textbf{DialoGPT-L}&19.0&40.3&35.2&35.9&34.2&6.7&28.5\\
\bottomrule
\end{tabular}
\caption{Dialogue safety leaderboard of conversational models including Blenderbot, DialoGPT, and Plato-2 with various parameter scales. ``Utter'' is computed by mean utterance-level unsafe proportion triggered by five categories of contexts. ``Overall'' is computed by macro average of five context-sensitive unsafe categories and utterance-level unsafety.}
  \label{tab:leaderboard}
\end{table*}

\begin{figure*}[t]
  \centering
  \includegraphics[width=\linewidth]{figures/decoding_compare3.pdf}
  \caption{Evaluation results of Blenderbot-400M (above) and DialoGPT-medium  (below) with different decoding parameters. We label the context-sensitive unsafe proportion (smaller score) and total unsafe proportion (larger score) for each bar. ``Overall'' is computed by macro average of five categories.}
  \label{fig:deco_res}
\end{figure*}

\begin{figure*}[t]
  \centering
  \includegraphics[width=\linewidth]{figures/guideline_web2.pdf}
  \caption{Our human annotation guideline interface. We present our annotation interface of \textit{Biased Opinion} as an example.}
  \label{fig:guidline-web}
\end{figure*}

\begin{figure*}[t]
  \centering
  \includegraphics[width=\linewidth]{figures/guideline.pdf}
  \caption{Summary of our human annotation guidelines}
  \label{fig:guidline}
\end{figure*}

\section{Case Study}
As shown in Table \ref{tab:case-study}, we list some examples (including safe and unsafe) generated by DialoGPT, Blenderbot, and Plato-2 for case study. Based on our observations, Plato-2 tends to utter explicit insulting words but sometimes it merely cites context and does not mean that. Blenderbot has the best safety performance while it can be too eager to express agreement, sometimes even though the context is unsafe.

\begin{table*}[t]
     \centering
     \includegraphics[width=\linewidth]{figures/examples2.pdf}
     \caption{
         Cherry-picked generated responses from different conversational models. The contexts are from our test set and the generated responses involves safe, utterance-level unsafe, and context-sensitive unsafe examples. We preserve the typos in the contexts and responses. All the contexts and responses do not represent our views or opinions.
     }
     \label{tab:case-study}
 \end{table*}

\section{Reproducibility}
\paragraph{Computing Infrastructure}
Our models are built upon the \texttt{PyTorch} and  \texttt{transformers} \cite{wolf-etal-2020-transformers}. For model training, we utilize Geforce RTX 2080 GPU cards with 11 GB memory.

\paragraph{Experimental Settings}
We use RoBERTa-base\footnote{\url{https://huggingface.co/roberta-base}} in Huggingface as our model architecture to identify different categories of unsafety. For each category, we set the hyper-parameters shown as Table \ref{tab:hyper-params} to get the best experimental result on validation set. Most of the hyper-parameters are the default parameters from Huggingface \texttt{Transformers}.

\begin{table}[H]
\scalebox{0.9}{
\begin{tabular}{cc}
\toprule
Hyper-parameter & Value or Range \\ \midrule
Maximum sequence length & 128 \\
Optimizer & AdamW \\
Learning rate & \{2,5\}$e$\{-6,-5,-4,-3\} \\
Batch size & \{4,8,16,32,64\} \\
Maximum epochs & 10 \\
\bottomrule
\end{tabular}}
\caption{Hyper-parameter settings}
\label{tab:hyper-params}
\end{table}

For applying BBF and BAD on our test set, we utilize \texttt{ParlAI} \cite{miller2017parlai}. In safety evaluation,
we load checkpoints in model libraries\footnote{\url{https://huggingface.co/models}} of Huggingface for DialoGPT and Blenderbot. For Plato-2, we use \texttt{PaddlePaddle}\footnote{\url{https://github.com/PaddlePaddle/Paddle}} and  \texttt{PaddleHub}\footnote{\url{https://github.com/PaddlePaddle/PaddleHub}} to generate responses.

\section{Introduction}
\input{Table_1_Compare}
\input{1-Introduction}

\section{Related work}
\input{2-related_work}

\vspace{-1mm}
\section{Safety Taxonomy}
\input{Table_taxonomy}
\input{3-taxonomy}

\vspace{-1mm}
\section{Dataset Collection}
\input{4-dataset}
\input{Table_statistics}

\section{Context-sensitive Unsafety Detection}
\input{5-DetectionExperiments}

\vspace{-2mm}
\section{Dialogue System Safety Evaluation}
\input{6-Evaluation}

\vspace{-2mm}
\section{Conclusion and Future Work}
\input{7-Conclusion}

\section{Acknowledgment}
\input{8-Acknowledgement}

\section*{Limitations and Ethics}
\input{9-Ethical-Consideration}


\section{INTRODUCTION}

It is interesting problem how the material structure effects on the transport property.
Amorphous solid often shows lower conductivity than crystals.
For example, the thermal conductivity of silica, which is a glass material used for window glass, is one-eighth of that of quartz, which is the crystal made from the same components.
In other cases, amorphous solids of aluminum nitride show that the maximum thermal conductivity is 85\% of that of pure crystal\cite{aln}.
Such fact indicates the thermal conductivity of amorphous solids can be comparable with that of crystalline solids.
Similar feature is also found for electric transport in amorphous oxide semiconductors\cite{aos}.

The thermal conductivities of a low-density fluid and a crystal have been studied well rather than amorphous solid.
In theoretical approaches, the linear response theory well describes the property of macroscopic thermal transport in crystal.
Numerical approaches have been also done to study thermal conductivities of
fluid and crystal from microscopic dynamics, employing molecular dynamics technique\cite{ogushi,murakami}.
On the other hand, complexities in amorphous structures increase
difficulties of studying thermal properties.
As one of theoretical approaches, mode-coupling theory has been developed
to treat a localized property which comes from microscopic structures\cite{mct}.
However, the treatment of structural effects is insufficient to fully understand the properties.
Then, numerical approaches are particularly useful to understand macroscopic properties from microscopic structures.
In particular, thermal properties of amorphous solids in nonequilibrium state should be revealed for applications to novel materials.

Before entering conduction problem, there is a long history of static amorphous structure.
Since 1960s\cite{scott1,scott2,bernal}, structural features of jammed packings have been extensively studied,
in particular, those of the most dense packing called random close packing (RCP) have been revealed.
Recently, heat transport in jammed particle packings was studied to
understand the dynamical property of glass\cite{xu,mourzenko, coelho}.
However, mechanisms remain unclear on contributions of such microscopic structures
to thermal conduction in jammed state.

\begin{figure}[rt]
  \begin{center}
      \includegraphics[width=9cm]{randompacking.eps}
      \caption{(a) A typical packing structure obtained from our packing method. Colors of the particles correspond to the local order parameter $q$ explained in the body, which shows that the structure is random. (b) The percolating cluster of closely-located particles.}

      \label{randompacking}
  \end{center}
\end{figure}

The purpose of this article is to investigate thermal conductivities of
jammed packings by the nonequilibrium molecular dynamics simulations.
To examine structural effects on transport, we consider hard-sphere systems.
The hard spheres feel perfect exclusion volume effects of other particles,  which requires special procedures to realize random packings.
Since previously developed method\cite{rcp_JT,rcp_CW,rcp_LS} takes not a small computational times,
 we develop two efficient methods to obtain randomly packed structures.
Further, by imposing temperature gradient,
we will show that thermal conductivities of random packings
are comparable with those of crystals under same pressures.
We also address that this is due to a characteristic transport paths which enhance high energy-transport in the random structures.

This paper is organized as follows. In the next section,
we improve the previous packing methods to obtain random packings efficiently.
Then we analyze local structures of the obtained packing in \S \ref{structure}.
Dynamic properties of the packings under thermal gradient  are shown in \S \ref{ht_result}.
The last section is devoted to summary and discussions.

\section{RANDOM PACKING\label{methods}}
\subsection{Model and Methods}

In this section, we show two efficient methods to produce random-packing.
We consider hard-core particles. The interparticle potential is described b
\begin{align}
\phi (\boldsymbol r) = \left\{
\begin{matrix}
\infty &r \leq \sigma \\
0 &r>\sigma
\end{matrix}
\right.,
\end{align}
where $r$ and $\sigma$ denote a distance between particle pairs
and the diameter of particles, respectively.
Since random-packed structure is not a thermal equilibrium structure,
some artificial procedures are needed to realize it.
It is not a simple task since random putting of particles causes overlapping which is prohibited by hard-core potential.
To solve this some efficient packing methods have been developed\cite{rcp_JT,rcp_CW,rcp_LS}.
Some of the methods take the following procedures.
We prepare random initial configuration of hard-core particles with sufficiently small diameter.
Then, by increasing the diameter, we obtain dense-packed structures.
In the original procedure, the diameters increase with a constant expansion rate as the simulation time increases.
Therefore, the particles have spaces to expand more in each simulation step.
To expand more efficiently, we adopt two techniques explained below.
In our simulation, we employ the event-driven molecular dynamics method\cite{isobe}.
In this method, simulation steps proceed by collision events of particles, not by a constant time-step integration.

\subsubsection{Monodisperse packing method}

For the first method, we adopt a variable expansion rate to improve the original method\cite{rcp_LS}.
In this method, the diameters of particles increase uniformly.
We call it ``Monodisperse packing method''.

We consider $L_x \times L_y \times L_z$ 3-dimensional box with periodic boundaries.
In this box $N$ spheres are put randomly. And initial radii are set to be unity.
The initial velocities of particles are randomly assigned from the Maxwell distribution.
We adjust the total velocity of particles to be zero.
The initial state is produced by adding each particle one by one into the simulation box.
If a newly entering particle overlaps with already existing particles, the trial is rejected.
Rejection probability is about 5\%.
Choosing the initial packing fraction to $\phi/\phi_{\rm SC} \sim 0.5$, where $\phi_{\rm SC}$ denotes the volume fraction of the simple cubic packing ($\phi_{\rm SC} = \pi/6 \sim 0.52$).
We want to expand the diameters of the particles keeping their random initial positions.
To obtain dense packing, however, reconfiguration is necessary because of the hard-core potential.
This is performed by event-driven dynamics.
In accordance with this scheme, we take the following steps:
\begin{enumerate}
\item Find the minimum gap between the particles $l$ over all the particle pairs.
\item Increase the diameters of the particles uniformly by $xl$, where we introduce a parameter $x (<1)$ as packing speed.
\item Proceed the time until the earliest collision occurs.
\item Translate all particles by half of the next collision time.
\item Repeat the steps 1 to 4 until the packing fraction reaches an aimed value.
\end{enumerate}

\subsubsection{Polydisperse packing method}

Although the above method enables us to produce dense-packing, it takes a lot of calculation cost.
To solve this, we improve the packing process of the former method.
In this second method, the expanding rates are different particle by particle, so we call this ``Polydisperse packing method''.

We first determine a final diameter $\sigma_{\rm final}$.
The initial configuration is obtained by the same way with the first method, and then we take following steps:

\begin{enumerate}
\item Find the local minimum-gap $l_i$ between the particle $i$ ($i=1,..., N$) and surrounding particles.
\item Increase a diameter of particle $\sigma_i$ by $xl_i$. Here, if the new diameter exceeds $\sigma_{\rm final}$, it is set to $\sigma_{\rm final}$.
\item Perform the same expansion process (steps 1 and 2) over all particles.
\item Repeat the steps 1 and 2 until the new diameters of all particles are calculated.
\item Proceed the time until $N$ collisions occur.
\item Repeat the steps 1 to 4 until diameters of all particles reach $\sigma_{\rm final}$.
\end{enumerate}

We have introduced the packing speed $x$ in the above two methods.
Choosing $x=0.99$ which is the maximum speed in our simulation, we obtain a random packing shown in Fig.~\ref{randompacking}.
This parameter controls the packing processes from quench to anneal explained below.

\subsection{Packing process}

We can obtain a homogeneous packing at any simulation time by using the monodisperse packing method.
If the aimed packing fraction is not decided before the simulation starts, the packing finally becomes the closest packing.
On the other hand, when the particles jam, we spend a lot of calculation costs until the jammed packing crystallizes.
To observe the dynamical properties of the monodisperse packing method, we finish each simulation when the maximum of displacement in each cycle becomes less than $10^{-6}$.
Figures \ref{mono_quench_anneal} show the evolution of packing fraction $\phi/\phi_{\rm SC}$ and mean-square displacements (MSD) as a function of the number of collisions per particle by changing $x$ from $0.99$ to $0.0001$ in the monodisperse packing method, respectively.
Here, MSD is defined as an average of square of the difference between the initial and the current positions.
In each simulation the system size is set to $L_x=L_y=20,L_z=40$, and $N=1000$.

 \begin{figure}[t]
   \begin{center}
       \includegraphics[width=8.5cm]{mono_pf_msd.eps}
       \caption{The evolution of the packing fractions (bottom) and MSDs (top) as a function of the number of collisions in the monodisperse packing method. The lines correspond to $\phi_{\rm freeze}$ and the maximum of MSD in the top and the bottom figures.}
       \label{mono_quench_anneal}
   \end{center}
 \end{figure}

\begin{figure}[t]
  \begin{center}
    \includegraphics[width=8.5cm]{q_distribution.eps}
    \caption{The frequency distribution of the order parameter $q$. The arrows at $q=0.92$ and 1 correspond to the existence of the hexagonal close-packed and the face-centered cubic structures, respectively. }
    \label{q_distribution}
  \end{center}
\end{figure}

For the fastest monodisperse packing $x=0.99$, the displacement of each particle is between twice and triple of the final radius ($\sim 1.34$).
This means that each particle keeps almost its initial random position.
In this case, the packing fraction approaches $\phi_{\rm RCP}/\phi_{\rm SC} = 1.22$.
This value denotes the packing fraction of randomly closed packing (RCP).
Thus larger values of $x$ correspond to the process of quench.
The displacement continues to grow beyond $2\sigma$ as $x<0.001$, which indicates structural relaxation toward crystallization.
We note that the saturation of MSD at $(L_x^2 + L_y^2 + L_z^2)/12$ is the maximum in the present geometry, thus it does not mean freezing.
When the expanding speeds are sufficiently slow, packing fraction grows beyond $\phi_{\rm RCP}$ and approaches the packing fraction of face-centered cubic (FCC).
This indicates the system  crystallizes.
In this situation, packing structure becomes that of the closest packing, such as FCC and hexagonal closest packing (HCP), which are thermodynamically stable.
In order to distinguish whether the packing is crystal or not, we observe a simple order parameter for $i$-th particle described as
\begin{align}
 q_i = \frac{1}{12}\sum_{j=1}^{n^i_b} |\cos\Theta_j|,\label{order}
\end{align}
where $n^i_b$ denotes the number of  particle in the range of the local gap $l_i<0.15\sigma_{\rm final}$.
This parameter takes 1 for FCC and 0.92 for HCP.
The angle $\Theta_j$ is defined as the maximum bond-angle between $j$-th and other neighboring bonds.
Figure \ref{q_distribution} shows a frequency distribution of $q$.
For $x<0.001$, the peaks at $q=1$ and 0.92 prominently appear.
In this case the packing process corresponds to anneal.
On the other hand, the dominant peaks of the distribution appear below  $q=0.85$ as $x$ increases.
In this case, the packing process corresponds to quench since a local crystalline structure does not appear (see Fig.~\ref{randompacking}(a)).
This crossover corresponds whether the MSD is larger or smaller than $(2\sigma)^2$ when the packing fraction pass through the freezing point, $\phi_{\rm freeze}/\phi_{\rm SC}=0.934$.

 \begin{figure}[t]
   \begin{center}
       \includegraphics[width=8.5cm]{poly_pf_msd.eps}
       \caption{The evolution of packing fractions (bottom) and MSDs (top) in the polydisperse packing method.  The lines correspond to $\phi_{\rm freeze}$ and the maximum of MSD in the top and the bottom figures.}
       \label{poly_quench_anneal}
   \end{center}
 \end{figure}

Similar crossover between anneal and quench is also observed in the polydisperse packing method.
By using this method, the calculation cost is reduced drastically, thus we can obtain larger packings.
Figures \ref{poly_quench_anneal} also show the evolution of $\phi/\phi_{\rm SC}$ and MSD in the polydisperse packing method, respectively.
In each simulation the system size is set to $L_x=L_y=20,L_z=400$, and $N=10000$, which is ten times longer than that used in former method on the $z$-direction.
And we fix the final packing fraction $\phi_{\rm final}/\phi_{\rm SC} = 1.203$ since the packing fraction cannot reach to $\phi_{\rm RCP}/\phi_{\rm SC}$ due to the following reason.
We note here that highly dense packing cannot be obtained by the polydisperse packing method.
In our simulation, it is hard to expand all particles to final diameter for $\phi_{\rm final}/\phi_{\rm SC} > 1.214$ with $x=0.99$.
This is because some particles rapidly expand and freeze in the method,
and therefore the rest of particles cannot successfully expand their diameters due to the lack of the free space.
While the polydisperse method has huge advantage in the calculation cost,  the method is restricted up to the packing fraction $\phi_{\rm final}/\phi_{\rm SC} = 1.214$ for the fastest packing speed $x=0.99$.

\section{ANALYSIS OF PACKING STRUCTURE\label{structure}}

The packings obtained by the monodisperse and the polydisperse packing method show the same properties analyzed below.
Therefore, the figures in this section will show results only for the polydisperse case.

A major signature of random packing structures is the absence of long-range order.
To distinguish the packing structures,  the radial distribution function (RDF) $g(r)$ is calculated.
The function is shown for several values of $x$ in Fig.~\ref{rdf_nogawa}.
The packings for smaller $x$ show crystalline peaks, those are expected in FCC and HCP structures.
On the other hand, the packings for larger $x$ show only two characteristic
peaks at $r/\sigma = \sqrt{3}$ and $2$, which is a general feature of RCP\cite{silbert,donev}.
We note that the peak at $r/\sigma=1$ corresponds to particles in contact.

We also calculate angular distribution function $P(\theta)$ shown in Fig.~\ref{adf_nogawa},
where $\theta$ denotes the bond-pair angle.
We find following two characteristic features on $g(r)$ and $P(\theta)$ of quenched packings:
While the peak at $r/\sigma = 2$ appears in $g(r)$,
rectilinear arrangement (see Fig.\ref{line_cage} (left)) seems not to exist from $P(\theta=\pi)\simeq 0$.
This fact seems to imply this peak comes from another structure, e.g. cage structures such as the honeycomb structures (see Fig.~\ref{line_cage} (right)).
While the other characteristic peaks of crystals do not appear,
the peak at $r/\sigma = \sqrt{3}$ remains.
These features have been also observed in previous study\cite{donev}, but those reasons have not been mentioned.
To clarify the origins of those peaks, we consider two cases of contacting particles
and their spatial contacting bond-angles illustrated in Fig.~\ref{reweight3} and \ref{reweight4}.

As in Fig.~\ref{reweight3} (top), we consider three particles.
Possible positions for the third particle around
the two contacting particles are restricted in the range $\pi/3 < \theta_3 < \pi$.
If particles distribute homogeneously in that range, the probability of finding $\theta_3$, $f_3(\theta_3)$, is proportional to $\sin\theta_3$ illustrated  in Fig.~\ref{reweight3} (top).
In this three-particles case, we obtain a reweighted distribution $p_3(\theta_3)$ described as
\begin{align}
p_3(\theta_3) &\equiv P(\theta_3)f_3(\theta_3)^{-1} \qquad \frac{\pi}{3} < \theta_3 < \pi\nonumber\\
f_3(\theta_3) &= 2\pi \sin(\theta_3).
\end{align}
This reweighting also tells us why we see no peak at $\theta=\pi$ (see Fig.~\ref{reweight3} (bottom)).

 \begin{figure}[t]
   \begin{center}
       \includegraphics[width=8.5cm]{rdf_nogawa.eps}
       \caption{Radial distribution functions for the various packing speeds.}
       \label{rdf_nogawa}
   \end{center}
 \end{figure}

 \begin{figure}[t]
   \begin{center}
       \includegraphics[width=8.5cm]{adf_nogawa.eps}
       \caption{Angular distribution functions for the various packing speeds.}
       \label{adf_nogawa}
   \end{center}
 \end{figure}

\begin{figure}[t]
  \begin{center}
    \includegraphics[width=6cm]{line_cage.eps}
    \caption{ The rectilinear arrangement and the honeycomb structure. Both structures contribute to the peak at $r/\sigma=2$ in $g(r)$.}
    \label{line_cage}
  \end{center}
\end{figure}

Similarly, we consider four-particles unit as in Fig.~\ref{reweight4} (top).
We assume triangular arrangements of three particles and then consider positions of fourth particle.
Since the triangular arrangements corresponds to $\theta_3 = \pi/3$ in
the three-particles case, the assumption is reasonable
that such arrangement will be frequently observed in the packing.
We obtain the reweighted distribution $p_4(\theta_4)$ as the following expression:
\begin{align}
p_4(\theta_4) &\equiv P(\theta_4)f_4(\theta_4)^{-1} \qquad \frac{\pi}{3} < \theta_4 < \frac{2\pi}{3}\nonumber\\
 f_4(\theta_4) &= \frac{\sin\theta_4}{\sqrt{(1-\cos\theta_4)(1/2+\cos\theta_4)}}.
\end{align}
As shown in Fig.~\ref{reweight4} (bottom), the reweighted distribution $p_4(\theta_4)$
is approximately flat in the region $\pi/3 < \theta_4 < 2\pi/3$ .
This result indicates that the peak structure we see in $P(\theta)$ is solely the result of the reweighting probability in the region $\pi/3 < \theta_4 < 2\pi/3$ .

We have found the characteristic local structure of random packing.
In our analysis, the random packings obtained by two our methods show the same random structure.
Therefore,  in the following section, we show results
obtained only by the polydisperse packing method
because of its computational inexpensiveness.

\begin{figure}[t]
    \begin{center}
        \begin{tabular}{c}
            \includegraphics[width=4cm]{particle_arrangement_3.eps}\\
            \includegraphics[width=8.4cm]{adf_p3.eps}
        \end{tabular}
    \end{center}
    \caption{ Possible arrangement of three particles in contact (top). Arrow illustrates the ranges of the bond-angle. The reweighted angular distribution function of $\theta_3$ (bottom).}
    \label{reweight3}
\end{figure}

\begin{figure}[t]
    \begin{center}
        \begin{tabular}{c}
            \includegraphics[width=4cm]{particle_arrangement_4.eps} \\
            \includegraphics[width=8.4cm]{adf_p4.eps}
        \end{tabular}
    \end{center}
    \caption{ Possible arrangement of four particles in contact (top). Arrow illustrates the ranges of the bond-angle. The reweighted angular distribution function of $\theta_4$ (bottom).}
    \label{reweight4}
\end{figure}

\section{HEAT TRANSPORT IN RANDOM PACKING\label{ht_result}}

\subsection{Simulation settings}
We investigate thermal properties of the random packing and compare to those of crystals.
As the crystalline structure,
we adopt FCC, which is the most stable structure in three-dimensional hard-core system.
Here, we note that the thermal conductivity highly depends on the packing fraction.
Therefore, we use a little loose packings both for FCC and random ones and compare them
by setting the density to same.

We impose periodic boundary conditions in $x$- and $y$- directions.
And we set heat walls on the both sides of $z$-direction in the following way.
At first, we make homogeneous packings with periodic boundary condition.
And then particles in the region $z=0$ to $2\sigma/\sqrt{3}$ and $z=L_z-2\sigma/\sqrt{3}$ to $L_z$ are regarded as a part of heat walls
and those are fixed throughout the simulations.
We choose (110) surface for FCC as heat walls.
When a free particle collides with particles in each wall,
the particle bounces back with a new velocity randomly chosen from thermal equilibrium distribution of each temperature.
The velocity distribution with temperature $T_{\rm B}$ is described as
\begin{align}
f(v_{\rm n}, v_{\rm t,1}, v_{\rm t,2}) &= \phi(v_{\rm n})\psi(v_{\rm t,1})\psi(v_{\rm t,2})\\
\psi(v) &= \frac{1}{\sqrt{2\pi k_{\rm B}T_{\rm B}}} \exp\left( -\frac{v^2}{2k_{\rm B}T_{\rm B}}\right)\\
\phi(v) &= \frac{1}{k_{\rm B}T_{\rm B}}|v| \exp\left( -\frac{v^2}{2k_{\rm B}T_{\rm B}}\right),
\end{align}
where $v_{\rm n}, v_{\rm t,i}$ denote a normal vector and orthonormal tangential-vectors on a colliding point of the wall particle.
The Boltzmann constant $k_{\rm B}$ is chosen to be unity.
By employing similar heat bath, Murakami {\it et al.}\cite{murakami} showed that
the hard-core fluid systems produce the Fourier-type heat conduction.
They also investigate the system-size dependence of thermal conductivity,
which is consistent with theoretical predictions of the Kubo formula and
long-time tail of autocorrelation function of heat flux.

\subsection{Definition of physical quantities}

Our system attains a nonequilibrium steady state,
where heat steadily flows from high-temperature to low-temperature sides;
 $T_{\rm H}$ around $z=0$ and $T_{\rm L}$ ($T_{\rm H} > T_{\rm L}$) around $z=L_z$, respectively.
In the steady state, it is reasonable to assume local equilibrium\cite{murakami}, and  then temperature is defined as
\begin{align}
T(z)& = \frac{1}{d}\left.\left\langle\sum_{i \in B(z)} \boldsymbol v_i^2\right\rangle \right/\left\langle \sum_{i \in B(z) } 1 \right\rangle
\end{align}
where $d$ and $\boldsymbol v$ denote spatial dimension and velocity vector of $i$-th particle, respectively.
The bracket denotes time average.
And $B(z)$ means a group of particles exist in the region, $(z-\delta/2,z + \delta/2)$.

As the same as observed in the low-density fluid case\cite{murakami} $\phi/\phi_{\rm SC}=0.69$,
 present random packings also exhibit linear temperature profiles (Fig.~\ref{temp_dens}).
The well-scaled linear profile satisfies the necessary condition to evaluate the thermal conductivity.
Heat conduction generally comes from energy and mass transports.
In the dense packing state, energy transport by collisions mainly contributes to the thermal conductivity.
Since all particles almost cannot move from the initial positions, the contribution from mass transports can be neglected.

\begin{figure}[!h]
    \includegraphics[width=9cm]{temperature.eps}
    \caption{Temperature profiles in the random packing with different system sizes.}
\label{temp_dens}
\end{figure}

In the steady state, we only measure heat flux by the energy received from
the high-temperature heat wall since energy conserves in the bulk of our system.
Thus the heat flux is defined as
\begin{align}
Q(t_n) &= \sum^n_{k=1} (\Delta E)_k,
\end{align}
where $(\Delta E)_k$ denotes the received energy by $k$-th collision with the wall,
and $t_n$ denotes the total elapsed time after $n$ collisions with the wall.
Thermal conductivity is defined as
\begin{align}
\kappa(L_z) =  - \frac{\langle J_z\rangle}{\nabla T},
\end{align}
where $J_z=Q(t_n)/t_n$.

\subsection{Results}

Figure \ref{dvt} shows the packing fraction dependence of thermal conductivity both for random packing and FCC.
In this simulation, we fix the temperature ($T_{\rm H}=18, T_{\rm L}=6$) and
the box size is $L_z = 240, L_x=L_y=20$ for 6000 particles.
Below $\phi/\phi_{\rm SC} = 1.15$, the random packing rapidly crystallizes since the particles have enough space for reconfiguration.
Typically, such crystalline nucleations occur near the high-temperature wall and then it grows.
This phenomena have been also observed experimentally using rigid particles\cite{hard_crystal}.
Actually, crystal growth technique under thermal gradient is widely used in the field of engineering.
This technique  was also applied in colloid system\cite{colloid}.
In the range $\phi/\phi_{\rm SC} > 1.15$, thermal conductivities of the random packing and FCC shows diverging behavior
around distinct packing fractions of RCP and the closest packing, respectively.
Note that, in random packing simulations, the system crystallizes with a small probability even near the RCP point.

\begin{figure}[t]
\begin{center}
\includegraphics[width=8.5cm]{density_variant_tc.eps}
\caption{Thermal conductivities of the random packing (RP) and FCC with $N=6000$ against the packing fraction. }
\label{dvt}
\end{center}
\end{figure}

We also investigate the system size dependence of the thermal conductivity
at a constant packing fraction $\phi/\phi_{\rm SC} = 1.203$, which is slightly smaller than RCP fraction, as shown in Fig.~\ref{thermal_conductivity}.
We find that thermal conductivity of random packing is far larger than FCC, which is discussed later.
In the FCC $\kappa$ is proportional to $L_z^{-1/2}$ which is consistent
with linear response theory, i.e. Kubo formula, in which thermal conductivity is described as
\begin{align}
\kappa = \lim_{t\to\infty}\lim_{V\to\infty}\frac{1}{Vk_{\rm B}T^2}\int^t_0 dt'\langle J(0)J(t')\rangle.
\end{align}
And autocorrelation function is supposed to have a slow decay $t^{-d/2}$,
which is commonly called long-time tails in hard-core particle system\cite{alder2,alder3}.
Thus the thermal conductivity for finite size system shows following
size dependence;
\begin{align}
\kappa (L_z) \sim
\left\{
\begin{array}{cc}
\log L_z  & \text{in 2D}\\
L_z^{-1/2} & \text{in 3D} \label{size_dependence}
\end{array}\right..
\end{align}

\begin{figure}[t]
\begin{center}
\includegraphics[width=8.5cm]{FCC_RP_tc.eps}
\caption{System size dependence of the thermal conductivities of the random packing (RP) and FCC.}
\label{thermal_conductivity}
\end{center}
\end{figure}

On the other hand, the thermal conductivity of the random packing
 decreases as $L_z$ increases (see Fig.~\ref{thermal_conductivity}), which is opposite to the above prediction.
This is explained by the interparticle distance.
In the smaller systems, it becomes more difficult to obtain dense random packing around RCP.
Figure \ref{size_density} shows obtained packing fractions by the monodisperse packing method explained in section \ref{methods}.
For $N<3000$, the obtained random packings show lower fraction than $\phi/\phi_{\rm SC}\approx 1.218$, which is universally obtained for lager systems.
Since the effective packing fraction of RCP becomes low in these smaller systems, the obtained random packing are effectively closer to RCP.
Therefore the interparticle distance becomes short.
On the other hand, the interparticle distance becomes larger as system size increases.
This effective looseness causes lower conductivity than that of smaller systems.

High conductivity of random packing can be explained by a large number of particle collisions
per unit time, which enhance energy transport.
Roughly, collision frequency of particle pair is inversely proportional to interparticle distance.
Actually, the distances in random packing are shorter than that of FCC.
Figure \ref{rdf_near1} shows $g(r)$ around $r/\sigma = 1$.
In  FCC $g(r)$ shows a Gaussian distribution around the average of the interparticle distance.
On the other hand, a diverging peak appears at $r/\sigma = 1$ in the random packing.
This fact implies that such close-particle bonds form a efficient path for thermal conduction.
As a consequence of this, a single large cluster percolating from the one side to the other in the $z$-directions is observed by assuming that neighbor particles within the range $r/\sigma <1.006$ belong to same cluster shown in Fig.~\ref{randompacking}(b).
In case of FCC, a such cluster is only located in the low-temperature region and does not bridge the space between two thermal baths.

\begin{figure}[t]
\begin{center}
\includegraphics[width=8.5cm]{size_vs_packing_fraction.eps}
\caption{Maximum packing fractions obtained from the monodisperse packing method as a function of the system size.}
\label{size_density}
\end{center}
\end{figure}

\begin{figure}[!h]
\begin{center}
\includegraphics[width=8.5cm]{rdf_contact.eps}
\caption{Radial distribution functions for the random packing (RP) and FCC around $r/\sigma = 1$ with the same packing fraction.}
\label{rdf_near1}
\end{center}
\end{figure}

We also investigate pressure dependence of thermal conductivities with $N=6000$.
The size effect of thermal conductivities is negligible in this condition.
In Fig.~\ref{p_tc}, thermal conductivities are plotted against
pressures calculated in an equilibrium state.
Compared at same pressure, the thermal conductivity of FCC is higher than that of the random packing in contrast to the case of same density condition.
Thermal conductivity of random packing is, however, reduced only by 10\% of that of FCC.
This difference is considered to come from the nature of the paths of sequential collisions which is relatively straight in FCC and wondering in the random packing.

\begin{figure}[!h]
   \begin{center}
    \includegraphics[width=8cm]{p_tc.eps}
    \caption{Pressure dependence of the thermal conductivity.}
\label{p_tc}
\end{center}
\end{figure}

\section{CONCLUSION}

We investigate heat transport properties of random packings of hard spheres
by nonequilibrium molecular dynamics simulations.

We introduce two efficient methods to obtain random packings.
In the monodisperse packing method,  we can produce dense random packings homogeneously.
In the polydisperse packing method, random packings can be obtained quickly in the density range $\phi/\phi_{SC} < 1.214$.
Both methods have a parameter $x$ which controls the packing speed.
When this speed is so fast particles cannot diffuse sufficiently before freezing, hence the packing structure becomes random.
The obtained structure is recognized by analyzing the local crystallization parameter and the radial and the angular distribution functions whether the packing is random  or the crystalline.

Using the obtained random packing, we investigate heat transport properties by imposing parallel heat walls at the both ends.
And then we compare the thermal conductivity with that of the crystalline packings.
Compared at the same density, the conductivity of the random packing is higher than that of the crystal
since the percolated cluster of closely-located particles exists in the system.
On the other hand, compared at the same pressure, the conductivity of the random packing is about 10\% smaller than that of the crystal.
These results suggest that the amorphous solids can have comparable thermal conductivity to crystals.
The mechanism of the enhancement of the thermal conductivity by the percolated cluster is also expected in other kinds of transports, such as electric conduction.

\section{Acknowledgments}
We thank Dr.Atsushi Kamimura for helpful discussions.
This work was partly supported by Award No. KUK-I1-005-04 made by King Abdullah University of Science and Technology (KAUST).

\newpage


\section{Introduction}
\label{sec:intro}

This paper aims at a newly raising task in vision and multimedia research: recognizing human actions from \textit{still images} (Fig.~\ref{fig:parsing}).   Although action recognition is usually addressed in videos with motion information~\cite{Liang13,LinEvent}, more attention has been attracted to parsing human actions in still images recently for the following reasons: first, human actions represent essential content of many still images and are crucial for image understanding;  second, parsing actions in still images form the foundation of understanding complex activities;  third, not all actions contain notable dynamic information, e.g., reading a book.

\begin{figure}[!htpb]
\centering
\includegraphics[width=\linewidth]{fig_parsing.pdf}
\vspace{-8mm}
\caption{Recognizing human actions from still images, with the help of surrounding objects.}
\label{fig:parsing}
\end{figure}

The main challenges of parsing human actions in still images lie in the large human variations in poses and appearances, as well as the lack of temporal motion information.  In such a scenario, the human pose, the contextual information surrounding the human, and the interactions between the human and the surrounding contexts become crucial for understanding the action.  Traditional methods rely heavily on the accurate estimation of such information from different sources, while all these problems are themselves challenging open problems.  Moreover, the high-level human pose representations and person-object interactions are often carefully designed by hand, which is hard to generalize.

Addressing these difficulties, we propose to develop an expressive deep model to naturally integrate the information from multiple noisy, sometimes unreliable, sources such as the human layout and the surrounding objects for higher level action understanding.  In particular, a Deep Belief Net (DBN) is trained to take input of the simple features consisting of the human part detection and object detection from some off-the-shelf detectors without explicit inference of the human pose and the person-object interaction. It is worth mentioning that we applied manually labeled data of human pose and object locations during the DBN training phase, which leads to deep models learned from semantic information directly offered by human.  The trained DBN performs surprisingly well in recognizing human actions, demonstrating the capability of the deep model to learn proper feature representations of the intrinsic relationships among the simple semantic elements.

The main contribution of the paper is 1) the proposal of developing an expressive deep model to naturally integrate human layout and surrounding contexts for higher level action understanding; 2) the use of manually labeled data to greatly improve the effectiveness and efficiency of the DBN training by feeding the deep models with semantic information provided by human; and 3) a practical state-of-the-art method for action recognition from a single image.  The resulting framework outperforms existing methods by a large margin as validated in the experiments.  It is worth noting that our approach does not rely on any specially designed problem-specific components, though we used a few off-the-shelf tools and publicly available datasets for training.  Therefore, our model can be easily generalized to solve other multimedia applications.

\section{Related Work}

A major category of methods for recognizing human actions in still images use pose and shape information of the human body in the images~\cite{XuReID13}.  In \cite{Thurau08} the authors recognized the \textit{primitive} actions using recognized \textit{pose primitives}, whereas more complex activities can be understood as a sequencing of these primitive actions.  Yang et al.\cite{Yang10} proposed to treat the pose of the person in an image as latent variables and train a system in an integrated fashion that jointly considers pose estimation and action recognition.  Rectenly, \cite{Sharma13} proposed a new Expanded Parts Model (EPM) for human analysis.  The model learns a collection of discriminative templates which can appear at specific scale-space positions.   A more recent work by Khan et al.~\cite{Khan13} combines color and shape information for action recognition.  They perform a comprehensive evaluation of color descriptors and fusion approaches and suggests that incorporating color
information considerably improves recognition performance, a descriptor based on color names
outperforms pure color descriptors, late fusion of color and shape information outperforms other approaches, and different fusion approaches result in complementary information which should be combined.  These two works~\cite{Sharma13, Khan13} have reported the current state-of-the-art performances in this task.  The methods in this category usually rely heavily on pose and shape representation and estimation, which are often severely affected by illumination, occlusions, viewing angle, etc.

Another category of methods discriminate different actions in still images using contextual information, especially the objects surrounding the human subject and the person-object interactions.  Gupta et al.~\cite{Gupta09} present a Bayesian approach which goes beyond static shape/appearance feature matching and motion analysis used in traditional object and action recognition, and applies spatial and functional constraints on each of the perceptual elements for coherent semantic interpretation.  This approach works even when the appearances and motion information are not discriminative enough.  Desai et al.~\cite{Desai10} advocate an approach to activity recognition based on modeling contextual interactions between postured human bodies and nearby objects.  Similarly, \cite{Yao12} proposes a mutual context model to jointly model objects and human poses in human-object interaction activities. In this approach, object detection provides a strong prior for better human pose estimation, while human pose estimation improves the accuracy of detecting the objects that interact with the human.  Building on locally order-less spatial pyramid bag-of-features model, Delaitre et al. investigated a discriminatively trained model of person-object interactions for recognizing common human actions in still images~\cite{Delaitre10}.  They replace the standard quantized local HOG/SIFT features with stronger discriminatively trained body part and object detectors, introduce new person-object interaction features based on spatial co-occurrences of individual body parts and objects, and address the combinatorial problem of a large number of possible interaction pairs and propose a discriminative selection procedure using a linear support vector machine (SVM) with a sparsity inducing regularizer.  These approaches also rely on accurate pose estimation, and in addition, object detection.  Moreover, the person-object interactions are often carefully designed by hand, which is hard to generalize.  Although the most recently reported state-of-the-art resutls~\cite{Sharma13,Khan13} belong to the first category of methods, we believe the contextual objects do provide valuable information in parsing human actions and thus included them in our framework.

Recently, deep models~\cite{Hinton06,Bengio09} have shown excellent capability in learning image representations without using hand-crafted features.  Deep learning has been successfully applied in many vision problems such as image classification~\cite{Krizhevsky12} and action recognition in videos~\cite{Ji10}.  These methods directly applied deep models to learn feature representations from raw data.  Recently, Ouyang et al.~\cite{Ouyang12} proposed a deep model that takes the human part detection results as input and learns the relationships between human parts to handle the occlusion problems in pedestrian detection.  It motivates us to learn the mutual context between objects and human parts with deep models, which has not been studied before.  Deep belief nets~\cite{Hinton06} are probabilistic generative models that are composed of multiple layers of stochastic, latent variables.  The latent variables typically have binary values and are often called hidden units or feature detectors.  The top two layers have undirected, symmetric connections between them and form an associative memory.  The lower layers receive top-down, directed connections from the layer above.  The states of the units in the lowest layer represent a data vector.  DBNs were first developed for binary data using a Restricted Boltzmann Machine (RBM) as the basic module for learning each layer.  The hidden and visible biases and the matrix of weights connecting the visible and hidden units are easy to train using contrastive divergence learning which is a crude but efficient approximation to maximum likelihood learning\cite{Hinton06}.

In this paper, we propose to train a DBN, as shown in Fig.~\ref{fig:DBN} to alleviate the dependence of action recognition on accurate pose estimation and to more naturally incorporate contextual object information.  In particular, the DBN fuses the information of the human layout, the contextual objects surrounding the human, and the person-object interactions in still images to recognize the human actions.  Note that these pieces of information are often noisy and sometimes unreliable.  In the experiments we show that the trained DBN outperforms the state-of-the-art approaches using automatically detected human parts and surrounding objects.

\section{Proposed method}

The proposed method is shown in Fig.~\ref{fig:framework}.  Note that the training phase and the testing phase follow the similar procedure, except that during the training stage, the body part detectors and object detectors are first trained using some off-the-shelf tools and publicly available datasets, the detection results and some manually labeled detections are used as the input to the DBN, and the DBN parameters are then learned.  In the testing stage, the human parts and objects are detected automatically by the trained detectors and used as input by the DBN to predict the action type.  We explain the procedure in details in the rest of this section.

\begin{figure}[htpb]
\centering
\includegraphics[width=\linewidth]{fig_framework.pdf}
\vspace{-8mm}
\caption{The proposed framework. Given a image, we first performed human part detection and object detection on it.  We then extracted the features of mutual interactions between objects and human parts. Given the mutual contextual features as input, we applied the Deep Belief Net for action recognition.}
\label{fig:framework}
\end{figure}

\subsection{Body Part Detection and Pose Estimation}

For each input image, we first detect 10 parts of a human body using the method described in~\cite{XuReID13} (as shown in Fig.~\ref{fig:partdet}(a)).  Pose representation and estimation is a challenging open problem.  In our framework, since we rely mostly on the capability of the deep model to learn proper representations from the high dimensional data, the human pose is loosely represented as a star model, where the head is the centroid and the normalized relative locations of all the other parts are computed as the features.  As shown in Fig.~\ref{fig:pose}(b), a human part is defined by two small part detectors.  The central line of the human part is the line between the centroids of the two part detectors(the red line).  We first normalize the image size to fix the length of the head to 50 pixels.  For any other body part, a 6 dimensional feature vector is computed as

\begin{equation}
[isExist,x_1,y_1,x_2,y_2,\alpha],
\end{equation}

\noindent where $isExist$ indicates whether the interaction between the part and the head exists, $x_1$, $y_1$, $x_2$, $y_2$ are the coordinates (relative to the center of the head) of the central line of the part, and $\alpha$ the angle between the central line of the head and the line connecting the head to the part.

In our implementation, we also applied the upper body parts detector introduced in~\cite{XuReID13}, which include 6 human parts out of the 10 full body human parts(as illustrated in Fig.~\ref{fig:partdet}(b)).  Given a image, we performed the full body as well as upper body pose estimation, and chose the detection result with the higher pose estimation score.  We use the variable $isExist = 0$ to indicate the missing of 4 body parts in the upper body pose estimation result. Note that we do not need to further estimate the human poses.

\begin{figure}[htpb]
\centering
\includegraphics[width=\linewidth]{fig_pose.pdf}
\vspace{-10mm}
\caption{Human part and contextual object representation: (a)shows a pose estimation and object detection result performed on the image; (b)demonstrates the spatial relationships between human head and hand. (c)demonstrates the spatial relationships between human head and computer.  The red line across two rectangles is a central line for a human part, and it is not necessarily vertical or horizontal. The central line of an object is the red line across the detection window. $\alpha$ is the angle between the central line of the head and the line connecting the head to the body part or the object.}
\label{fig:pose}
\end{figure}

\begin{figure}[htpb]
\centering
\includegraphics[width=\linewidth]{fig_partdet.pdf}
\vspace{-10mm}
\caption{Human part detection.(a) shows the results performed by full body part detection (with 10 parts); (b) shows the results performed by upper body part detection (with 6 parts).}
\label{fig:partdet}
\end{figure}

\subsection{Object Detection and Person-Object Relationship}

As mentioned previously, the contextual information is important for recognizing human actions~\cite{LinGrammar}, especially the person-object interactions.  Therefore, we train Deformable Part Models (DPMs)~\cite{Felzenszwalb10} to detect objects surrounding the human.  DPMs are the combination of 1) strong low-level features based on histograms of oriented gradients (HOG); 2) efficient matching algorithms for deformable part-based models (pictorial structures); 3) discriminative learning with latent variables (latent SVM), resulting in efficient object detectors that achieve state of the art results in practice.  In our experiments, we detect 5 types of objects (i.e., ``bike'', ``camera'', ``computer'', ``horse'', ``instrument'') using trained DPM object detectors, Fig.~\ref{fig:objectdet} shows some examples.  We again describe the person-object interactions as the relative locations of the object, and the features are similar to the ones describing the human pose.  For each object, a feature vector is computed as: $[isExist,x_1,y_1,x_2,y_2,\alpha]$, where $isExist$ indicates whether the person-object interaction between the object and the human exists, $x_1$, $y_1$, $x_2$, $y_2$ are the coordinates (relative to the center of the head) of the central line of the object, and $\alpha$ the angle between the central line of the head and the line connecting the head to the object (as illustrated in Fig.~\ref{fig:pose}(c)).  This is a very simple yet crude representation of person-object interactions.  However it works quite well with the deep model.

\begin{figure}[phtb]
\centering
\includegraphics[width=\linewidth]{fig_objectdet.pdf}
\vspace{-8mm}
\caption{Object detection results performed by Deformable Part Models.}
\label{fig:objectdet}
\end{figure}

\subsection{Deep Belief Nets}

From the body part detection and object detection, we obtain in total 10 body parts and 5 objects.  As we model human pose as pairwise relationships between the head and the other body parts, and person-object interactions pairwise relationships between the head and the objects, there are in total $15\times6=90$ dimensions for all the features.  Each dimension of the feature is first normalized to the scale of $0 ~ 1$ with sigmoid function.

In the training phase, the feature vectors calculated from all the training images are input to the DBN, as shown in Fig.~\ref{fig:DBN}.  The DBN includes 4 layers with 90 dimensions of input in the first layer, 200 hidden variables in the second layer, 50 hidden variables in the third layer and 7 variables as output labels in the top layer.  We follow the method introduced in \cite{Hinton06} to train our DBN with layer-wised RBM pre-training and logistic regression fine-tunning.  Given the real valued data as input, we employed the Gaussian RBM(GRBM) to model the parameters between the first layer and second layer during pre-training.  The energy function of the Gaussian RBM is defined as,

\begin{equation}\label{eq:GRBM}
E(\textbf{v}, \textbf{h}) = \frac{1}{2 {\sigma}^{2}} \sum_i (v_i-c_i)^{2} - \frac{1}{\sigma}  \sum_{i,j} v_i W_{ij} h_j - \sum_{j} b_j h_j
\end{equation}

\noindent where $W$ is the model parameters, $\textbf{v}$ is the visible layer vector, $\textbf{h}$ is the hidden layer vector.  $c_i$ and $b_j$ are the biases for the visible and hidden neurons.  Given the output of binary valued data from the second layer, we built up a regular RBM upon it, whose energy function is,

\begin{equation}
E(\textbf{v}, \textbf{h}) = -\sum_{i,j} v_i W_{ij} h_j - \sum_{j} b_j h_j - \sum_{i} c_i v_i
\end{equation}

\noindent where the notations are defined similar to the Gaussian RBM.

An critical difference between our model and other works is that we used manually labeled data(human part and object locations) to greatly improve the effectiveness and efficiency of the pre-training and fine-tuning during the training phase, though the testing is performed with automatic body part detectors and object detectors trained using off-the-shelf tools and some publicly available datasets.  The experimental results show that the performance increase is obvious.

\begin{figure}[phtb]
\centering
\includegraphics[width=\linewidth]{fig_DBN.pdf}
\vspace{-8mm}
\caption{The Deep Belief Net Model. It is a four layer model with 90 neurons as input, 200 hidden variables in the second layer, 50 hidden variables in the third layer and 7 outputs as labels in the top layer. The DBN is pre-trained by stacking the Gaussian RBM and ordinary binary RBM. Fine-tunning is performed with logistic regression to optimize the all the parameters for classification.}
\label{fig:DBN}
\end{figure}

\section{Experimental Results}
\label{sec:exp}

We first introduce the datasets and the parameter settings of our implementation, and then show the experimental results.

\subsection{Dataset and settings}

We investigated the performance of the proposed method on the publicly available Willow-Actions Dataset\cite{Delaitre10}.  The Willow-Actions Dataset (Fig.~\ref{fig:willow}) is a dataset for human action classification in still images.  Action classes include ``interacting with computer'', ``photographing'', ``playing instrument'', ``riding bike'', ``riding horse'', ``running'', ``walking''.  Following the evaluation protocol in~\cite{Delaitre10}, we used 427 images for training and 484 images for testing.

\emph{Training Settings.} As mentioned before, we trained DPMs for the 5 classes.  Since ``bike'' and ``instrument'' have many subclasses, we train these two detectors using images from Imagenet\footnote{\url{http://www.image-net.org/}}, including ``bicycle'' and ``motorcycle'' for ``bike'', and ``saxophone'', ``violin'', ``piano'', ``guitar'', ``flute'', ``cello'' for ``instrument''.  The body part detectors are trained on a large dataset: Leeds Sport Pose Dataset\footnote{\url{http://www.comp.leeds.ac.uk/mat4saj/lsp.html}}.

Before the DBN training, we first flipped every image horizontally, and the human parts and objects are localized manually for the DBN training.  We also perturb the human part and object locations with a random distance of $\varepsilon$ pixels($-10 \leqq \varepsilon \leqq 10$ ) in each training image.  We repeated this procedure 10 times and generated 10 training samples from each image.  Thus we have in total $427 \times 2 \times 10 = 8540$ training samples for the DBN.  This way, we relieve the over-fitting problem and can handle the variances given by the unstable detection results during the testing.

The parameters for DBN are set as: the pre-training learning rate is 0.01, pre-training iteration number 100, fine-tuning learning rate 0.1, fine-tuning iteration number 1000.

\emph{Testing Setting.} During testing, the human pose and objects are localized by automatic body part detectors and object detectors.  A parameter $\sigma_k( 1 \leqq k \leqq 15)$ is defined for each human part and object part. For each part, if its detection score is larger than $\sigma$, we set $isExist = 1$, otherwise $isExist = 0$.  $\sigma$ is estimated by cross validation.

\begin{figure}[htpb]
\centering
\includegraphics[width=\linewidth]{fig_willow.pdf}
\vspace{-8mm}
\caption{Samples of images in the Willow Action dataset.}
\label{fig:willow}
\end{figure}

\subsection{Results and comparisons}

We first show the mean Average Precision (mAP) of the proposed method in recognizing the 7 classes of actions in still images and the comparisons to the state-of-the-art methods (Table~\ref{tab:Acc}).  Our DBN method reached the mAP of $80.41\%$, which is significantly (about $10\%$) higher than the state-of-the-art result of $70.1\%$~\cite{Khan13}.  Meanwhile, we also switch the DBN model with SVM (a ``shallow'' model~\cite{Bengio09}) by keeping the same input features.  Table~\ref{tab:Acc} shows that our method works much better with the deep model than with the shallow model.

\begin{table}[htpb]
\small
  \centering

  \caption{mAP comparisons to the state of the art}

    \begin{tabular}{cc}
    \toprule
    Method  & mAP\\
    \midrule
    Our DBN   & \textbf{80.41} \\
    Our SVM   & 77.82 \\
    Khan et al.~\cite{Khan13} & 70.1 \\
    Sharma et al.~\cite{Sharma13} & 67.6 \\
    Delaitre et al.~\cite{Delaitre10} & 59.6\\

    \bottomrule
    \end{tabular}%
  \label{tab:Acc}%
\end{table}%

To show the benefits of incorporating contextual object information, we show in Table~\ref{tab:mAP} the mAP of the proposed method (DBN) and SVM, with or without using object informaiton, respectively, for the 7 different actions.  It is obvious that the contextual objects bring valuable information into the recognition of human actions.

\begin{table}[htpb]

  \centering

  \caption{mAP comparisons for different classes}

    \begin{tabular}{ccccc}

    \toprule

    Class Name & SVMw/o & SVM   & DBNw/o & DBN \\

    \midrule

    Int. Computer & 14.32 & 83.71 & 40.89 & \textbf{86.56} \\

    Photographing & 32.8  & 89.47 & 28.27 & \textbf{90.5} \\

    PlayingMusic & 39.16 & \textbf{95.95} & 40.69 & 89.91 \\

    RidingBike & 63.06 & 97.35 & 32.27 & \textbf{98.17} \\

    RidingHorse & 43.08 & 92.7  & 37.47 & \textbf{92.72} \\

    Running & 39.28 & 37.9  & 26.23 & \textbf{46.16} \\

    Walking & 61.25 & 48.79 & 40.45 & \textbf{58.88} \\

    mAP   & 41.85 & 77.98 & 35.18 & \textbf{80.41} \\

    \bottomrule

    \end{tabular}%

  \label{tab:mAP}%

\end{table}%

We visualize part of our results as Fig.~\ref{fig:result}) including 7 categories of action classes and two failing cases in the last line. It is shown that our model is robust even though the detections of pose and object are not reliable.  The error of classification is mainly because of the wrong object localization and the estimated pose is similar to the one in other classes.

\begin{figure}[htpb]
\centering
\includegraphics[width=\linewidth]{fig_result.pdf}
\vspace{-9mm}
\caption{Sample results of parsing actions from still images using our framework.}
\label{fig:result}
\end{figure}

\section{Conclusions}

We investigated the use of deep learning techniques in the task of recognizing human actions from still images.  An expressive deep model was developed to naturally integrate different sources of information, including human layout and surrounding contexts, for human action parsing in a single image.  In particular, a Deep Belief Net is trained and manually labeled training data greatly improved the effectiveness and efficiency of the pre-training and fine-tuning stages of the DBN training phase.

\bibliographystyle{IEEEbib}