\section{Introduction}
Surveillance cameras has spread rapidly in most of the cities all around the world. Among the surveillance tasks computer vision has focused on tracking and detection of targets where targets are described by their visual appearance.
Nevertheless, recently the task of capturing as many people characteristics as possible has gained importance for better understanding a situation and its attendants. The task, referred in literature as \emph{attribute recognition} \cite{rap}, consists in detecting people attributes (such as age, sex, etc.) and items (backpacks, bags, etc.) of people through security cameras.
While this task have been profitably attacked from a face recognition perspective capturing gender, age, and race,\cite{other3,other2}, very few works focus on whole people body. Among these, most of them, \cite{other8,pedestrian4}, consider people always unoccluded and at full resolution that is not the case when dealing with surveillance footages. In fact, surveillance cameras, that have typically a far field of view, are massively affected by resolution issues and people occlusion, Fig. \ref{fig:1}. \\In this work we propose an attribute recognition method that explicitly deals with resolution and occlusions by exploiting a generative deep network approach \cite{dcgan}. Our proposal consists of three deep networks. The first classifies people attributes given full body images. The others focus on enhancing the input image by raising its resolution and trying to reconstruct images from occlusion by means of a generative convolutional approach \cite{gan_sr2}. To our knowledge, this is the first work that considers this task in a surveillance context by explicitly dealing with those both issues.

\begin{figure}[t]
\begin{center}
   \includegraphics[width=.9\columnwidth]{src/front}
\end{center}
   \caption{Resolution and occlusions issues and reconstructed frames by our generative approach.}
   \label{fig:1}
\label{fig:front}
\end{figure}

\begin{figure*}
\begin{center}	\includegraphics[scale=0.70]{src/arch1}
\end{center}
   \caption{Architecture of our Attribute Classification Network.}
\label{fig:class}
\end{figure*}
\section{Related Work}
Early works on attribute recognition usually treat attributes by independently training a different classifier for each attribute, \cite{pedestrian1,pedestrian2}. More recently Convolutional Neural Networks (CNN), enable researchers to mine the relationship between attributes and are preferred on large scale object recognition problems because of their advanced performances. There are large bodies of work on CNNs, like \cite{other3} which undertakes the task of occlusion and low-resolution robust facial gender classification, or \cite{other2,other4} that predict facial attributes from faces in the wild. Many other works like \cite{other6,other7} propose different methods to achieve attribute classification like gender, smile and age in an unconstrained environment. However, those technique involve only facial images and are not suitable for surveillance tasks.
Moreover \cite{other8} addresses the problem of describing people based on clothing attributes. Nevertheless, our work considers the person as a whole and does not focus only on clothing classification.
More recent works that rely on full-body images to infer human attributes are the Attribute Convolutional Net (ACN) and Deep learning based Multi-Attribute joint Recognition model (DeepMAR), \cite{acn,pedestrian4}.
ACN jointly learns different attributes through a jointly-trained holistic CNN model, while DeepMAR utilizes the prior knowledge in the object topology for attribute recognition. In \cite{part_cnn1, part_cnn3} attributes classification is accomplished by combining part-based models and deep learning by training pose-normalized CNNs. Additionally, MLCNN \cite{pedestrian3} splits the human body in 15 parts and train a CNN for each of them while DeepMAR* \cite{rap} divides the whole body in three parts which correspond to the headshoulder part, upper body and lower body of a pedestrian respectively. Furthermore, \cite{context} tackles the problem of attribute recognition by improving a part-based method within a deep hierarchical context.
Nevertheless, the majority of those methods relies on high resolution images and does not encompass the problem of occlusion. Recent works on image super-resolution exploit Generative Adversarial Networks (GAN) \cite{gan}, and more precisely Deep Convolutional Adversarial Networks (DCGAN) \cite{dcgan}, in order to generate high resolution images starting from low resolution ones \cite{gan_sr1, gan_sr2}.

\begin{figure*}
\begin{center}
	\includegraphics[scale=0.54]{src/arch2}
\end{center}
   \caption{Architecture of our Reconstruction Network.}
\label{fig:gen}
\end{figure*}

\section{Method}
The contribution of this work consists on three networks: a baseline state-of-the-art part-based architecture for human attribute classification based on ResNet \cite{resnet}, a generative model that aims to reconstruct the missing body parts of people in occluded images and a second generative model that is capable of enhance the resolution of images at low resolution.

\subsection{Attribute Classification Network}
The proposed approach for human attribute classification is inspired by the previous part-based works thus capable of learning pose-normalized deep feature representations from different parts. By blending together the capability of neural networks with a part-based approach grants robustness when facing unconstrained images dominated by the effects of pose and viewpoints.
Inspired by \cite{context}, we propose to decompose the input image $I$ into blocks which correspond to the whole body $b$ and a set of parts $\{ p \in P \}$, (inputs in Fig \ref{fig:class}). We choose three parts: head-shoulder section, upper body and lower body of the pedestrian. Those four blocks are then passed through the ResNet50 network \cite{resnet} pretrained on ImageNet classification \cite{imnet} to obtain four part-based convolutional feature maps. Note that, in order to achieve this, we replaced the last average pooling $7 \times 7$ in ResNet50 with a global average pooling. This allowed us to fed the network with images that have one of their dimension smaller than $227$. After computing the feature maps, we branch out two attribute score paths. On the first path, we use the full body feature maps in order to obtain a prediction score based on the whole person. We incorporate the part-based technique in the second path where we compute a prediction score for each image partition (scores in Fig \ref{fig:class}), followed by a $\max$ score operation that aims to select the most descriptive part for each attribute. This operation is needed because human attributes often reside in a specific body area (e.g. "hat" is in the head-shoulder part). The final attribute prediction is performed by adding the whole body score to the part score:
\begin{multline}
\label{eq:eg}
	Score_i(I) = Score_i(b) + Score_i(P) \\
	= w_{i,b}^T \cdot \phi(b) + \max_{p \in P}  w_{i,p}^T \cdot \phi(p)
\end{multline}
where $w_{i,\cdot}$ are the scoring weights of the $i$th attribute for different regions, while $\phi(\cdot)$ are the feature maps from different regions.

The whole network, depicted in Fig. \ref{fig:class}, is trained using a \textit{weighted binary cross entropy} loss with a different weight for each attribute. The need of using a weighted loss arise from the fact that the distributions of attributes classes in the dataset may be imbalanced:
\begin{multline}
\label{eq:eg}
	Loss_{C} = -\sum_{i=1}^A \frac{1}{2 r_i} \cdot y_i \cdot \log( \hat{y_i}) \\
	+ \frac{1}{2(1 - r_i)} \cdot (1 - y_i) \cdot \log(1 - \hat{y_i})
\end{multline}
where A is the total number of attributes, $y$ is the ground truth label vector and $\hat{y}$ is the predicted label vector. For each attribute, $r$ is the ratio between the number of samples that hold that attribute and the total number of samples.
\subsection{Reconstruction Network}
\label{subsec:reconstruction}
Our second goal consist in making the previous architecture robust to occlusion integrating a system capable of removing the obstructions and replacing them with body parts that could likely belong to the occluded person. In the worst case scenario, even though the replaced body parts does not reflect the real attributes of the person, the reconstruction still helps the classifier: by removing the occlusion we produce an image that contains only the subject without noise that could lead to misclassifications. For example, an image containing a person occluded by another person could induce the network to classify the attributes of the person in the foreground which is not the subject of the image.
In order to accomplish this, we train a generative function $G^R$ capable of estimating a reconstructed image $I^R$ from an occluded input image $I^{O}$. During training $I^{O}$ is obtained by artificially partially overlapping an image $I$ with another image. To achieve our goal we train a generator network as a feed-forward CNN $G^R_{{\theta}_g}$ with parameters ${\theta}_g$. For $N$ training images we solve:
\begin{equation}
\label{eq:eg}
	\hat{\theta}_g = \arg \min_{{\theta}_g} \frac{1}{N} \sum_{n=1}^N Loss_R(G^R_{{\theta}_g}(I^{O}_n), I_n)
\end{equation}
Here $\hat{\theta}_g$ is obtained by minimizing the loss function $Loss_{R}$ described at the end of this section.

Following \cite{gan}, we further define a discriminator network $D^R_{{\theta}_d}$ with parameters ${\theta}_d$ that we train alongside with $G^R_{{\theta}_g}$ in order to solve the adversarial min-max problem:
\begin{multline}
\label{eq:eg}
	\min_{G^R} \max_{D^R} \mathbb{E}_{I \sim p_{data}(I)}[\log{D^R(I)}] \\
	+ \mathbb{E}_{I^{O} \sim p_{gen}(I^{O})}[\log{1 - D^R(G^R(I^{O}))}]
\end{multline}
The purpose of the discriminator $D^R$ is to distinguish generated images from real images, meanwhile the generator $G^R$ is trained with the aim of fooling the discriminator $D^R$. With this approach we obtain a generator model capable of learning solution that are similar to not occluded images thus indistinguishable by the discriminator.
Inspired by \cite{dcgan} we propose the generator's architecture illustrated in Figure~\ref{fig:gen}. Specifically, in the encoder we use four strided convolutional layers (with stride 2) to reduce the image resolution each time the number of feature is doubled, SConvs in Fig. \ref{fig:gen}. The decoding uses four transposed convolutional layers (also known as fractionally strided convolutional layers) to increase the resolution each time the number of feature is halved, and a final convolution, TConvs in Fig. \ref{fig:gen}. We use Leaky ReLU as activation function in the encoding phase and ReLU in the decoding phase. We adopt batch-normalization layers before activations (except for the last Conv) and a kernel size $5 \times 5$ at each step.
The discriminator architecture is similar to the generator's encoder except for the number of filter, which increase by a factor of 2 from 128 to 1024. The resulting 1024 feature maps are followed by one sigmoid activation function in order to obtain a probability useful for the classification problem. We use batch-normalization before every Leaky ReLU activation, except for the first layer.

The definition of the loss function $Loss_{R}$ is fundamental for the effectiveness of our generator network. Borrowing the idea from \cite{gan_sr2}, we propose a loss composed by a weighted combination of two components:
\begin{equation}
\label{eq:eg}
	Loss_R = Loss_{SSE} + \lambda Loss_{gen}
\end{equation}
Here $Loss_{SSE}$ is the reconstruction loss based on sum of squared errors of prediction (SSE) which let the generator predict images that are pixel-wise similar to the target image. The pixel-wise SSE is calculated between downsized versions of the generated and target images, first applying an averaged pooling layer. This is because we want to avoid the standard "blurred" effect that MSE and SSE trained autoencoders suffer from. In our experiments we used a $\lambda$ equals to $10^{-1}$.

The second component $Loss_{gen}$ is the actual adversarial loss of the generator $G^R$ which encourages the network to generate perceptually good solutions that are in the subspace of person-like images. The loss is defined as follows:
\begin{equation}
\label{eq:eg}
	Loss_{gen} = \sum_{i=1}^N \log(1 - D^R(G^R(I^{O})))
\end{equation}
Where $D^R(G^R(I^{O}))$ is the probability of the discriminator labeling the generated image $G^R(I^{O})$ as being a real image.

\subsection{Super Resolution Network}
\label{subsec:multires}
Our last goal is to integrate our system with a network capable of enhancing the quality of images that have poor resolution.
This task is accomplished by training another generative function $G^S$ capable of estimating an high resolution image $I^{H}$ from a low resolution input image $I^{L}$. During training $I^{L}$ is obtained from the original image $I$ by performing a simple downsample operation with factor $r = 4$. To achieve our goal we train the generator network as a feed-forward CNN likewise we did for $G^R$.
As for the Reconstruction Network, we define the discriminator $D^S$ in the same way we defined $D^R$. For $D^S$ we used the same architecture used in $D^R$.
The architectural differences between the two models reside in the number of layers: in the Super Resolution Network we used three strided convolution (in the Encoder), with 256, 512 and 1024 features respectively and five transposed convolutions (in the Decoder) that follow the pattern 512, 256, 256, 128, 128.  The motivation is that the input image in $G^S$ is two time smaller with respect to the input image in $G^R$.
Moreover in $D^S$ we used five strided convolution with the number of filter that increase by a factor of 2 from 128 to 2048.
Eventually we set the $\lambda$ value used to weight the loss components to $1$.
\section{Experiments}
We conduct our experiments using the new RAP \cite{rap} dataset, a very richly annotated dataset with 41,585 pedestrian samples, each of which is annotated with 72 attributes as well as viewpoints, occlusions and body parts information. As recommended by \cite{rap}, we measure performances with five metrics: mean accuracy (mA), accuracy, precision, recall and F1 score.
We perform three type of experiments. First, we comparative evaluated the performances of our Attribute Classification Network comparing the results with other deep SoA approaches. Secondly, we corrupted the dataset with occlusions (occRAP $2^{nd}$ row in Fig. \ref{fig:rap})and tested the benefits achieved by the combination of our Reconstruction Network, Sec. \ref{subsec:reconstruction} with the Attribute Classification one. Thirdly, we corrupt the dataset by lowering the resolution (lowRAP $3^{rd}$ row in Fig \ref{fig:rap}) and evaluate the contribution of our Superesolution Network, Sec. \ref{subsec:multires}, in conjuction with Attribute Classification. Eventually, we propose a complete classification pipeline where all the three network are combined together on both low-res and occluded images.
\begin{figure}
\begin{center}
	\includegraphics[width=\columnwidth]{src/rap}
\end{center}
   \caption{First row original RAP dataset. Second row occlusion RAP dataset where the images have been occluded from 50 to 80\% by pasting the upper body of another subject. Third row low resolution RAP where the image are downsamples of a factor 4 on width and height respectively.}

\label{fig:rap}
\end{figure}
The occRAP dataset is produced by randomly overlapping RAP images in order to artificially reproduce the occlusions. Note that in our experiment we focused the attention only on one type of occlusion: the occlusion that cover the bottom part of an image where the occluded portion have been randomly sampled from 50\% to 80\% occlusion rate. lowRap, instead, is obtained by performing a simple downsample with factor 4 from the original RAP images.
\paragraph{Attribute Classification}
Following \cite{rap} we conducted the experiments on the RAP dataset with 5 random splits. For each split, totally 33,268 images are used for training and the rest 8,317 images are used for testing. Due to the unbalanced distribution of attributes in RAP we selected the 50 attributes that have the positive example ratio in the dataset higher than 0.01. For each image we also add one attribute corresponding to the occlusion of our interest (\emph{occlusion down} attribute).
For each mini-batch, we resized the images to a fixed dimension of $320 \times 128$. In order to split the figure in the three parts we divide the height in 10 blocks and pick the top 4 for the head-shoulder part, the third to the seventh for the upper body part and the sixth to the tenth for the lower body part. The network is trained using stochastic gradient descent with a learning rate $10^{-5}$, learning rate decay $10^{-6}$ and momentum $0.9$. We used 8 images per mini-batch.
Tab.~\ref{tab:comparison} shows the results on RAP dataset where our baseline is compared against state-of-the-art methods on the same 51 attributes \footnote{Complete per-attribute results are in the supplementary material.}.
It can be shown that our network perform favourably in terms of Accuracy being competitive in both Precision and Recall related metrics. This is mainly due to the adoption of the fixed body part partitions of the image that, in case of people images from surveillance cameras, represent a reliable partition of the body in its parts. Additionally, parts scoring maximization allow for selecting the most reliable score for every individual attribute thus increasing the classification accuracy.
\paragraph{Reconstruction}
We trained our Reconstruction Network with the occRAP training set and simultaneously providing the network with the original not occluded images associated to the inputs in order to compute the $Loss_{SSE}$. For optimization we used Adam with $\beta_1 = 0.5$ and learning rate of $0.002$. We alternate updates to the discriminator and generator network with $K = 1$ as recommended in \cite{gan}.\\
Furthermore, the aim is to quantify the impact that occluded images have in the classification task. We firstly fed our classification network with images picked from occRAP testing set obtaining the results reported in Tab.\ref{tab:curruptExp} while visual examples are depicted in Fig. \ref{fig:reco}. Secondly we repeated the experiment manipulating the input images using our Reconstruction Network with the aim of removing the occlusion. In the same table are reported the results that shows a significant improvement. From the results, it emerges that our Reconstruction Network provide a reasonable guess of the occluded person appearance being able of learning from the visible part a potentially useful image completion.

\begin{table}
\footnotesize
\begin{center}
\begin{tabular}{l|c|c|c|c|c}

Method & mA & Accuracy & Precision & Recall & F1 \\
\hline\hline
ACN \cite{acn}      		& 69.66 & 62.61 & \bfseries 80.12 & 72.26 & 75.98 \\
DeepMAR \cite{pedestrian4} 	& 73.79 & 62.02 & 74.92 & 76.21 & 75.56 \\
DeepMAR* \cite{rap} 		& 74.44 & 63.67 & 76.53 & 77.47 &  77.00 \\
\hline
Our      & \bfseries 79.73 & \bfseries 83.97 & 76.96 & \bfseries 78.72 & \bfseries 77.83 \\
\hline
\end{tabular}
\end{center}
\caption{Comparison with SoA on the RAP dataset.}
\label{tab:comparison}
\end{table}

\begin{table}
\footnotesize
\begin{center}
\begin{tabular}{l|c|c|c|c|c}

Input & mA & Accuracy & Precision & Recall & F1 \\
\hline\hline
\multicolumn{6}{c}{\bf{occRap experiment}}\\
occRAP & 57.70 & 61.00 & 33.26 & 41.63 & 33.25 \\
\hline
occRAP + NET &\bfseries 68.81 & \bfseries 74.54 & \bfseries 57.29 & \bfseries 58.91 & \bfseries 58.09 \\
\hline
\multicolumn{6}{c}{\bf{lowRap experiment}}\\
lowRAP & 63.80 & 74.51 & 44.47 & 49.56 & 40.37 \\
\hline
lowRAP + NET &\bfseries 76.02 & \bfseries 80.12 & \bfseries 69.56 & \bfseries 73.12 & \bfseries 71.30 \\
\hline
\multicolumn{6}{c}{\bf{Complete Experiment}}\\
\hline\hline
Corrupted &60.68 & 72.75 & 38.67 & 45.47 & 41.80 \\
\hline
Restored & \bfseries 65.82 & \bfseries 76.01 & \bfseries 48.98 & \bfseries 55.50 & \bfseries 52.04 \\
\end{tabular}
\end{center}
\caption{Experiments with occlusions (occRAP experiment), low resolution (lowRAP experiment) and Complete Model. The complete experiment uses the merge of test sets occRAP and lowRAP. \emph{Corrupted} are the score of the Attribute Classification network on plain input data. \emph{Restored} are the results when using our complete pipeline.}
\label{tab:curruptExp}
\end{table}

\begin{figure}
\begin{center}
	\includegraphics[width=\columnwidth]{src/occlusion}
\end{center}
   \caption{Visual examples of Generative Reconstruction. For every triplet of images: (Leftmost)the original image; (Middle)Occluded input; (Rightmost)Reconstruction/Guessed unoccluded image.}
\label{fig:reco}
\end{figure}

\paragraph{Resolution}
The dataset used for training the Super Resolution Network is the lowRAP. We trained the network with the original size images associated to the inputs in order to compute the $Loss_{SSE}$. We adopted the same optimizer and the same $K$ value used for the Reconstruiction Network.
In order to evaluate how the low resolution affects the performance on attributes classification, we inputed the lowRAP test set to our Attribute Classification Network, after a 4x bilinear upsampling. Subsequently, we passed the images through our Super Resolution Network before attribute classification. As can be seen in Tab.~\ref{tab:curruptExp} the adoption of our Super Resolution network leads to an important improvement being able to keep more information w.r.t. the upsampling.
\paragraph{Complete Model}
Our final experiment consists in testing all our networks in order to build a system that is able to detect corrupted images and consequently react performing a restoring operation when possible.
To achieve this we propose a simple algorithm where the input is passed through the Super Resolution Network only if the input image is smaller than the network input. The image is then passed through the Classification Network and, if the \emph{occlusion down} attribute is positively triggered (the test F1 score of the \emph{occlusion down} attribute is $>85\%$), the image is passed through the Reconstruction Network. The reconstructed image is finally fed again to the Classification Network to output the final scores. The test were performed on the merge of occRAP and lowRAP test sets.  Tab.~\ref{tab:curruptExp} highlights the improved results obtained by this pipeline w.r.t. our Deep Attribute Classification Network alone.
\section{Conclusions}
In this work we presented the use of Deep Generative Network for image enhancing in people attributes classification. Our Generative Network have been designed to overcome two common problems in surveillance scenarios, namely people resolution and occlusions. Experiments have shown that jointly enhancing images before feeding them to an attribute classification network can improve the results even when input images are affected by those issues.
In further works we will explore the fusion of the networks in a single end-to-end model that can automatically choose which enhancement network activates by looking at images at test time. We find this line of work can foster research about the problem of attribute classification in surveillance contexts where camera resolution and positioning cannot be neglected.
\section{Acknowledgements}
The work is supported by the Italian MIUR, Ministry of Education, Universities and Research, under the project COSMOS Prin 2015 programme.
{\small
\bibliographystyle{ieee}

\section{Introduction}
Surveillance cameras has spread rapidly in most of the cities all around the world. Among the surveillance tasks computer vision has focused on tracking and detection of targets where targets are described by their visual appearance.
Nevertheless, recently the task of capturing as many people characteristics as possible has gained importance for better understanding a situation and its attendants. The task, referred in literature as \emph{attribute recognition} \cite{rap}, consists in detecting people attributes (such as age, sex, etc.) and items (backpacks, bags, etc.) of people through security cameras.
While this task have been profitably attacked from a face recognition perspective capturing gender, age, and race,\cite{other3,other2}, very few works focus on whole people body. Among these, most of them, \cite{other8,pedestrian4}, consider people always unoccluded and at full resolution that is not the case when dealing with surveillance footages. In fact, surveillance cameras, that have typically a far field of view, are massively affected by resolution issues and people occlusion, Fig. \ref{fig:1}. \\In this work we propose an attribute recognition method that explicitly deals with resolution and occlusions by exploiting a generative deep network approach \cite{dcgan}. Our proposal consists of three deep networks. The first classifies people attributes given full body images. The others focus on enhancing the input image by raising its resolution and trying to reconstruct images from occlusion by means of a generative convolutional approach \cite{gan_sr2}. To our knowledge, this is the first work that considers this task in a surveillance context by explicitly dealing with those both issues.

\begin{figure}[t]
\begin{center}
   \includegraphics[width=.9\columnwidth]{src/front}
\end{center}
   \caption{Resolution and occlusions issues and reconstructed frames by our generative approach.}
   \label{fig:1}
\label{fig:front}
\end{figure}

\begin{figure*}
\begin{center}	\includegraphics[scale=0.70]{src/arch1}
\end{center}
   \caption{Architecture of our Attribute Classification Network.}
\label{fig:class}
\end{figure*}
\section{Related Work}
Early works on attribute recognition usually treat attributes by independently training a different classifier for each attribute, \cite{pedestrian1,pedestrian2}. More recently Convolutional Neural Networks (CNN), enable researchers to mine the relationship between attributes and are preferred on large scale object recognition problems because of their advanced performances. There are large bodies of work on CNNs, like \cite{other3} which undertakes the task of occlusion and low-resolution robust facial gender classification, or \cite{other2,other4} that predict facial attributes from faces in the wild. Many other works like \cite{other6,other7} propose different methods to achieve attribute classification like gender, smile and age in an unconstrained environment. However, those technique involve only facial images and are not suitable for surveillance tasks.
Moreover \cite{other8} addresses the problem of describing people based on clothing attributes. Nevertheless, our work considers the person as a whole and does not focus only on clothing classification.
More recent works that rely on full-body images to infer human attributes are the Attribute Convolutional Net (ACN) and Deep learning based Multi-Attribute joint Recognition model (DeepMAR), \cite{acn,pedestrian4}.
ACN jointly learns different attributes through a jointly-trained holistic CNN model, while DeepMAR utilizes the prior knowledge in the object topology for attribute recognition. In \cite{part_cnn1, part_cnn3} attributes classification is accomplished by combining part-based models and deep learning by training pose-normalized CNNs. Additionally, MLCNN \cite{pedestrian3} splits the human body in 15 parts and train a CNN for each of them while DeepMAR* \cite{rap} divides the whole body in three parts which correspond to the headshoulder part, upper body and lower body of a pedestrian respectively. Furthermore, \cite{context} tackles the problem of attribute recognition by improving a part-based method within a deep hierarchical context.
Nevertheless, the majority of those methods relies on high resolution images and does not encompass the problem of occlusion. Recent works on image super-resolution exploit Generative Adversarial Networks (GAN) \cite{gan}, and more precisely Deep Convolutional Adversarial Networks (DCGAN) \cite{dcgan}, in order to generate high resolution images starting from low resolution ones \cite{gan_sr1, gan_sr2}.

\begin{figure*}
\begin{center}
	\includegraphics[scale=0.54]{src/arch2}
\end{center}
   \caption{Architecture of our Reconstruction Network.}
\label{fig:gen}
\end{figure*}

\section{Method}
The contribution of this work consists on three networks: a baseline state-of-the-art part-based architecture for human attribute classification based on ResNet \cite{resnet}, a generative model that aims to reconstruct the missing body parts of people in occluded images and a second generative model that is capable of enhance the resolution of images at low resolution.

\subsection{Attribute Classification Network}
The proposed approach for human attribute classification is inspired by the previous part-based works thus capable of learning pose-normalized deep feature representations from different parts. By blending together the capability of neural networks with a part-based approach grants robustness when facing unconstrained images dominated by the effects of pose and viewpoints.
Inspired by \cite{context}, we propose to decompose the input image $I$ into blocks which correspond to the whole body $b$ and a set of parts $\{ p \in P \}$, (inputs in Fig \ref{fig:class}). We choose three parts: head-shoulder section, upper body and lower body of the pedestrian. Those four blocks are then passed through the ResNet50 network \cite{resnet} pretrained on ImageNet classification \cite{imnet} to obtain four part-based convolutional feature maps. Note that, in order to achieve this, we replaced the last average pooling $7 \times 7$ in ResNet50 with a global average pooling. This allowed us to fed the network with images that have one of their dimension smaller than $227$. After computing the feature maps, we branch out two attribute score paths. On the first path, we use the full body feature maps in order to obtain a prediction score based on the whole person. We incorporate the part-based technique in the second path where we compute a prediction score for each image partition (scores in Fig \ref{fig:class}), followed by a $\max$ score operation that aims to select the most descriptive part for each attribute. This operation is needed because human attributes often reside in a specific body area (e.g. "hat" is in the head-shoulder part). The final attribute prediction is performed by adding the whole body score to the part score:
\begin{multline}
\label{eq:eg}
	Score_i(I) = Score_i(b) + Score_i(P) \\
	= w_{i,b}^T \cdot \phi(b) + \max_{p \in P}  w_{i,p}^T \cdot \phi(p)
\end{multline}
where $w_{i,\cdot}$ are the scoring weights of the $i$th attribute for different regions, while $\phi(\cdot)$ are the feature maps from different regions.

The whole network, depicted in Fig. \ref{fig:class}, is trained using a \textit{weighted binary cross entropy} loss with a different weight for each attribute. The need of using a weighted loss arise from the fact that the distributions of attributes classes in the dataset may be imbalanced:
\begin{multline}
\label{eq:eg}
	Loss_{C} = -\sum_{i=1}^A \frac{1}{2 r_i} \cdot y_i \cdot \log( \hat{y_i}) \\
	+ \frac{1}{2(1 - r_i)} \cdot (1 - y_i) \cdot \log(1 - \hat{y_i})
\end{multline}
where A is the total number of attributes, $y$ is the ground truth label vector and $\hat{y}$ is the predicted label vector. For each attribute, $r$ is the ratio between the number of samples that hold that attribute and the total number of samples.
\subsection{Reconstruction Network}
\label{subsec:reconstruction}
Our second goal consist in making the previous architecture robust to occlusion integrating a system capable of removing the obstructions and replacing them with body parts that could likely belong to the occluded person. In the worst case scenario, even though the replaced body parts does not reflect the real attributes of the person, the reconstruction still helps the classifier: by removing the occlusion we produce an image that contains only the subject without noise that could lead to misclassifications. For example, an image containing a person occluded by another person could induce the network to classify the attributes of the person in the foreground which is not the subject of the image.
In order to accomplish this, we train a generative function $G^R$ capable of estimating a reconstructed image $I^R$ from an occluded input image $I^{O}$. During training $I^{O}$ is obtained by artificially partially overlapping an image $I$ with another image. To achieve our goal we train a generator network as a feed-forward CNN $G^R_{{\theta}_g}$ with parameters ${\theta}_g$. For $N$ training images we solve:
\begin{equation}
\label{eq:eg}
	\hat{\theta}_g = \arg \min_{{\theta}_g} \frac{1}{N} \sum_{n=1}^N Loss_R(G^R_{{\theta}_g}(I^{O}_n), I_n)
\end{equation}
Here $\hat{\theta}_g$ is obtained by minimizing the loss function $Loss_{R}$ described at the end of this section.

Following \cite{gan}, we further define a discriminator network $D^R_{{\theta}_d}$ with parameters ${\theta}_d$ that we train alongside with $G^R_{{\theta}_g}$ in order to solve the adversarial min-max problem:
\begin{multline}
\label{eq:eg}
	\min_{G^R} \max_{D^R} \mathbb{E}_{I \sim p_{data}(I)}[\log{D^R(I)}] \\
	+ \mathbb{E}_{I^{O} \sim p_{gen}(I^{O})}[\log{1 - D^R(G^R(I^{O}))}]
\end{multline}
The purpose of the discriminator $D^R$ is to distinguish generated images from real images, meanwhile the generator $G^R$ is trained with the aim of fooling the discriminator $D^R$. With this approach we obtain a generator model capable of learning solution that are similar to not occluded images thus indistinguishable by the discriminator.
Inspired by \cite{dcgan} we propose the generator's architecture illustrated in Figure~\ref{fig:gen}. Specifically, in the encoder we use four strided convolutional layers (with stride 2) to reduce the image resolution each time the number of feature is doubled, SConvs in Fig. \ref{fig:gen}. The decoding uses four transposed convolutional layers (also known as fractionally strided convolutional layers) to increase the resolution each time the number of feature is halved, and a final convolution, TConvs in Fig. \ref{fig:gen}. We use Leaky ReLU as activation function in the encoding phase and ReLU in the decoding phase. We adopt batch-normalization layers before activations (except for the last Conv) and a kernel size $5 \times 5$ at each step.
The discriminator architecture is similar to the generator's encoder except for the number of filter, which increase by a factor of 2 from 128 to 1024. The resulting 1024 feature maps are followed by one sigmoid activation function in order to obtain a probability useful for the classification problem. We use batch-normalization before every Leaky ReLU activation, except for the first layer.

The definition of the loss function $Loss_{R}$ is fundamental for the effectiveness of our generator network. Borrowing the idea from \cite{gan_sr2}, we propose a loss composed by a weighted combination of two components:
\begin{equation}
\label{eq:eg}
	Loss_R = Loss_{SSE} + \lambda Loss_{gen}
\end{equation}
Here $Loss_{SSE}$ is the reconstruction loss based on sum of squared errors of prediction (SSE) which let the generator predict images that are pixel-wise similar to the target image. The pixel-wise SSE is calculated between downsized versions of the generated and target images, first applying an averaged pooling layer. This is because we want to avoid the standard "blurred" effect that MSE and SSE trained autoencoders suffer from. In our experiments we used a $\lambda$ equals to $10^{-1}$.

The second component $Loss_{gen}$ is the actual adversarial loss of the generator $G^R$ which encourages the network to generate perceptually good solutions that are in the subspace of person-like images. The loss is defined as follows:
\begin{equation}
\label{eq:eg}
	Loss_{gen} = \sum_{i=1}^N \log(1 - D^R(G^R(I^{O})))
\end{equation}
Where $D^R(G^R(I^{O}))$ is the probability of the discriminator labeling the generated image $G^R(I^{O})$ as being a real image.

\subsection{Super Resolution Network}
\label{subsec:multires}
Our last goal is to integrate our system with a network capable of enhancing the quality of images that have poor resolution.
This task is accomplished by training another generative function $G^S$ capable of estimating an high resolution image $I^{H}$ from a low resolution input image $I^{L}$. During training $I^{L}$ is obtained from the original image $I$ by performing a simple downsample operation with factor $r = 4$. To achieve our goal we train the generator network as a feed-forward CNN likewise we did for $G^R$.
As for the Reconstruction Network, we define the discriminator $D^S$ in the same way we defined $D^R$. For $D^S$ we used the same architecture used in $D^R$.
The architectural differences between the two models reside in the number of layers: in the Super Resolution Network we used three strided convolution (in the Encoder), with 256, 512 and 1024 features respectively and five transposed convolutions (in the Decoder) that follow the pattern 512, 256, 256, 128, 128.  The motivation is that the input image in $G^S$ is two time smaller with respect to the input image in $G^R$.
Moreover in $D^S$ we used five strided convolution with the number of filter that increase by a factor of 2 from 128 to 2048.
Eventually we set the $\lambda$ value used to weight the loss components to $1$.
\section{Experiments}
We conduct our experiments using the new RAP \cite{rap} dataset, a very richly annotated dataset with 41,585 pedestrian samples, each of which is annotated with 72 attributes as well as viewpoints, occlusions and body parts information. As recommended by \cite{rap}, we measure performances with five metrics: mean accuracy (mA), accuracy, precision, recall and F1 score.
We perform three type of experiments. First, we comparative evaluated the performances of our Attribute Classification Network comparing the results with other deep SoA approaches. Secondly, we corrupted the dataset with occlusions (occRAP $2^{nd}$ row in Fig. \ref{fig:rap})and tested the benefits achieved by the combination of our Reconstruction Network, Sec. \ref{subsec:reconstruction} with the Attribute Classification one. Thirdly, we corrupt the dataset by lowering the resolution (lowRAP $3^{rd}$ row in Fig \ref{fig:rap}) and evaluate the contribution of our Superesolution Network, Sec. \ref{subsec:multires}, in conjuction with Attribute Classification. Eventually, we propose a complete classification pipeline where all the three network are combined together on both low-res and occluded images.
\begin{figure}
\begin{center}
	\includegraphics[width=\columnwidth]{src/rap}
\end{center}
   \caption{First row original RAP dataset. Second row occlusion RAP dataset where the images have been occluded from 50 to 80\% by pasting the upper body of another subject. Third row low resolution RAP where the image are downsamples of a factor 4 on width and height respectively.}

\label{fig:rap}
\end{figure}
The occRAP dataset is produced by randomly overlapping RAP images in order to artificially reproduce the occlusions. Note that in our experiment we focused the attention only on one type of occlusion: the occlusion that cover the bottom part of an image where the occluded portion have been randomly sampled from 50\% to 80\% occlusion rate. lowRap, instead, is obtained by performing a simple downsample with factor 4 from the original RAP images.
\paragraph{Attribute Classification}
Following \cite{rap} we conducted the experiments on the RAP dataset with 5 random splits. For each split, totally 33,268 images are used for training and the rest 8,317 images are used for testing. Due to the unbalanced distribution of attributes in RAP we selected the 50 attributes that have the positive example ratio in the dataset higher than 0.01. For each image we also add one attribute corresponding to the occlusion of our interest (\emph{occlusion down} attribute).
For each mini-batch, we resized the images to a fixed dimension of $320 \times 128$. In order to split the figure in the three parts we divide the height in 10 blocks and pick the top 4 for the head-shoulder part, the third to the seventh for the upper body part and the sixth to the tenth for the lower body part. The network is trained using stochastic gradient descent with a learning rate $10^{-5}$, learning rate decay $10^{-6}$ and momentum $0.9$. We used 8 images per mini-batch.
Tab.~\ref{tab:comparison} shows the results on RAP dataset where our baseline is compared against state-of-the-art methods on the same 51 attributes \footnote{Complete per-attribute results are in the supplementary material.}.
It can be shown that our network perform favourably in terms of Accuracy being competitive in both Precision and Recall related metrics. This is mainly due to the adoption of the fixed body part partitions of the image that, in case of people images from surveillance cameras, represent a reliable partition of the body in its parts. Additionally, parts scoring maximization allow for selecting the most reliable score for every individual attribute thus increasing the classification accuracy.
\paragraph{Reconstruction}
We trained our Reconstruction Network with the occRAP training set and simultaneously providing the network with the original not occluded images associated to the inputs in order to compute the $Loss_{SSE}$. For optimization we used Adam with $\beta_1 = 0.5$ and learning rate of $0.002$. We alternate updates to the discriminator and generator network with $K = 1$ as recommended in \cite{gan}.\\
Furthermore, the aim is to quantify the impact that occluded images have in the classification task. We firstly fed our classification network with images picked from occRAP testing set obtaining the results reported in Tab.\ref{tab:curruptExp} while visual examples are depicted in Fig. \ref{fig:reco}. Secondly we repeated the experiment manipulating the input images using our Reconstruction Network with the aim of removing the occlusion. In the same table are reported the results that shows a significant improvement. From the results, it emerges that our Reconstruction Network provide a reasonable guess of the occluded person appearance being able of learning from the visible part a potentially useful image completion.

\begin{table}
\footnotesize
\begin{center}
\begin{tabular}{l|c|c|c|c|c}

Method & mA & Accuracy & Precision & Recall & F1 \\
\hline\hline
ACN \cite{acn}      		& 69.66 & 62.61 & \bfseries 80.12 & 72.26 & 75.98 \\
DeepMAR \cite{pedestrian4} 	& 73.79 & 62.02 & 74.92 & 76.21 & 75.56 \\
DeepMAR* \cite{rap} 		& 74.44 & 63.67 & 76.53 & 77.47 &  77.00 \\
\hline
Our      & \bfseries 79.73 & \bfseries 83.97 & 76.96 & \bfseries 78.72 & \bfseries 77.83 \\
\hline
\end{tabular}
\end{center}
\caption{Comparison with SoA on the RAP dataset.}
\label{tab:comparison}
\end{table}

\begin{table}
\footnotesize
\begin{center}
\begin{tabular}{l|c|c|c|c|c}

Input & mA & Accuracy & Precision & Recall & F1 \\
\hline\hline
\multicolumn{6}{c}{\bf{occRap experiment}}\\
occRAP & 57.70 & 61.00 & 33.26 & 41.63 & 33.25 \\
\hline
occRAP + NET &\bfseries 68.81 & \bfseries 74.54 & \bfseries 57.29 & \bfseries 58.91 & \bfseries 58.09 \\
\hline
\multicolumn{6}{c}{\bf{lowRap experiment}}\\
lowRAP & 63.80 & 74.51 & 44.47 & 49.56 & 40.37 \\
\hline
lowRAP + NET &\bfseries 76.02 & \bfseries 80.12 & \bfseries 69.56 & \bfseries 73.12 & \bfseries 71.30 \\
\hline
\multicolumn{6}{c}{\bf{Complete Experiment}}\\
\hline\hline
Corrupted &60.68 & 72.75 & 38.67 & 45.47 & 41.80 \\
\hline
Restored & \bfseries 65.82 & \bfseries 76.01 & \bfseries 48.98 & \bfseries 55.50 & \bfseries 52.04 \\
\end{tabular}
\end{center}
\caption{Experiments with occlusions (occRAP experiment), low resolution (lowRAP experiment) and Complete Model. The complete experiment uses the merge of test sets occRAP and lowRAP. \emph{Corrupted} are the score of the Attribute Classification network on plain input data. \emph{Restored} are the results when using our complete pipeline.}
\label{tab:curruptExp}
\end{table}

\begin{figure}
\begin{center}
	\includegraphics[width=\columnwidth]{src/occlusion}
\end{center}
   \caption{Visual examples of Generative Reconstruction. For every triplet of images: (Leftmost)the original image; (Middle)Occluded input; (Rightmost)Reconstruction/Guessed unoccluded image.}
\label{fig:reco}
\end{figure}

\paragraph{Resolution}
The dataset used for training the Super Resolution Network is the lowRAP. We trained the network with the original size images associated to the inputs in order to compute the $Loss_{SSE}$. We adopted the same optimizer and the same $K$ value used for the Reconstruiction Network.
In order to evaluate how the low resolution affects the performance on attributes classification, we inputed the lowRAP test set to our Attribute Classification Network, after a 4x bilinear upsampling. Subsequently, we passed the images through our Super Resolution Network before attribute classification. As can be seen in Tab.~\ref{tab:curruptExp} the adoption of our Super Resolution network leads to an important improvement being able to keep more information w.r.t. the upsampling.
\paragraph{Complete Model}
Our final experiment consists in testing all our networks in order to build a system that is able to detect corrupted images and consequently react performing a restoring operation when possible.
To achieve this we propose a simple algorithm where the input is passed through the Super Resolution Network only if the input image is smaller than the network input. The image is then passed through the Classification Network and, if the \emph{occlusion down} attribute is positively triggered (the test F1 score of the \emph{occlusion down} attribute is $>85\%$), the image is passed through the Reconstruction Network. The reconstructed image is finally fed again to the Classification Network to output the final scores. The test were performed on the merge of occRAP and lowRAP test sets.  Tab.~\ref{tab:curruptExp} highlights the improved results obtained by this pipeline w.r.t. our Deep Attribute Classification Network alone.
\section{Conclusions}
In this work we presented the use of Deep Generative Network for image enhancing in people attributes classification. Our Generative Network have been designed to overcome two common problems in surveillance scenarios, namely people resolution and occlusions. Experiments have shown that jointly enhancing images before feeding them to an attribute classification network can improve the results even when input images are affected by those issues.
In further works we will explore the fusion of the networks in a single end-to-end model that can automatically choose which enhancement network activates by looking at images at test time. We find this line of work can foster research about the problem of attribute classification in surveillance contexts where camera resolution and positioning cannot be neglected.
\section{Acknowledgements}
The work is supported by the Italian MIUR, Ministry of Education, Universities and Research, under the project COSMOS Prin 2015 programme.
{\small
\bibliographystyle{ieee}


\section{Introduction}
With the continuous development of mobile communication technology and the rapid development of mobile Internet, mobile terminals represented by smart phones, tablet computers, laptop, and smart assistants have been widely used. But the mobile terminal receives limiting factors such as volume, weight, performance, power, etc. Its working ability is still in a serious and tedious state, which cannot meet the increasing demand of people. Although the mobile terminal has made great progress in hardware technology (for example, the continuous replacement of CPU/GPU, the continuous improvement of chip manufacturing process from 28nm to 14nm to the current 7nm, 5nm\cite{8776580}, etc.), but it is still far from what people need. Moreover, with the emergence of new concepts such as autonomous driving, telemedicine, and Industry 4.0 which need ultra reliability, low latency\cite{7847322}, ordinary equipment is even more unable to support their operations. Meanwhile, with the emergence of machine learning, artificial intelligence and other emerging technologies\cite{7951770}, the rapid development of image recognition, speech recognition and other applications, virtual reality and augmented reality game applications are emerging in endlessly. The operation of these applications requires a large amount of computing resources and storage resources, and they are all computationally intensive applications at a time. Due to the limitation of  mobile terminals or some other devices, when computationally intensive applications\cite{7951770}  are running on smart terminals, the endurance of the terminal and the performance of the application are very problematic. How to solve this problem of resource limitation and energy consumption has become a huge challenge today.

Most of the literature suggests to change the task allocation method, transmission power, clock frequency to optimize the task offloading algorithm. In \cite{8279411}--\cite{8638800}, authors proposed task splitting, which is a way to change the  task structure to reduce the latency and improve the local device energy efficiency. However, they only split the task, but did not consider the redundancy of sources. In this paper,  we propose to split task to the smallest executable task, named as unit, and then select the unit by using the correlation between them. Furthermore, both time and task domain correlation is considered in our work to selected the necessary tasks to be processed.

\section{Problem Formulation and System Models}
In this section, the definition of task and unit will be introduced first to help to understand the proposed models. Both task and unit are the process of collecting data, processing data, and sending instructions, but unit is the smallest section that can form a task, and unit cannot be divided anymore. That means task can be composed of one or more units, and a task can be split into one or more units.

Consider $N$  devices (users), denoted by a set of  $\mathbb{N}=\{1, 2, ...., N\}$, and device $i$ has $M_i$ tasks at the same time,  $i\in \mathbb{N}$, denoted by a set of $\mathbb{M}=\{M_1, M_2, ...., M_N\}$, each task in the $M_i$ tasks is composed of $M_s$ different unit, denoted by a set of $\mathbb{K}=\{k_{M_1},k_ {M_2}, ...., k_{M_s}\}$and $i\in \mathbb{N}$. After some tasks are split, some identical units may be generated, so the correlation of task domain came into being. In our work, it is proposed to use the correlation between units to improve the energy-efficient of local device and reduce the latency. Moreover, it is assumed here that devices communicate with MEC server orthogonally. The main target here is to improve the energy efficiency of local devices when fulfil the extremely severe latency requirements of each unit and each device.

The energy cost  minimization problem is formulated as:
\begin{equation}\label{eqn01}
OPT-1\ \ \ \ \ \ \ \ \ \min \limits_{\mathcal{A,\ F,\ P}}\sum_{n=1}^{n=N}E_n^L
\end{equation}
Subject to{\color{black}{
\begin{equation}
		\begin{split}
C1: LT_{n,j}^m&\leq T_{n,j\ max},\ \ \ \ \ j \in A, n\in \mathbb{N}\\
C2: LT_{n,k}^l&\leq T_{n,k\ max},\ \ \ \ \ k \in B, n\in \mathbb{N}\\
C3: LT_{n}&\leq T_{n\ max},\ \ \ \ \  n\in\mathbb{N}\\
C4:f_{n,l}&\leq f_{n,l\ max},\ \ \ \ \ \ n\in \mathbb{N}\\
C5:p^t_{n}&\leq p^t_{n\ max}, \ \ \ \ \ \ n\in \mathbb{N}
		\end{split}
	\end{equation}}}

$E_n^L$ represents the local energy consumption of user $n$, $ LT_{n,j}^m$ represents the latency of unit $j$ of user n processed on MEC,  $LT_{n,k}^l$ represents the latency of unit k of user $n$ processed on local device, $ T_{n,j\ max}$ represents the latency requirement of unit $j$,  $A$ represents the collection of all tasks processed on the MEC, and $B$ represents the collection of all units processed on the local device, $LT_n$  represents the latency of user $n$,  $T_{n\ max}$ represents the latency requirement  of user $n$,  $f_{n,l}$   represents the computation capability of user $n$, such as the number of CPU cycles per second,  $ p^t_{n}$ represents the transmission power of  user $n$, and $f_{n,l\ max},\  p^t_{n\ max}$ indicate the maximum value of $f_{n,l}$ and $ p^t_{n}$ respectively.

 $\mathcal{F}=\{f_{n,l}|n\in \mathbb{N}\}$,  $\mathcal{P}=\{p^t_{n}|n\in \mathbb{N}\}$, $\mathcal{A}= \{A_{n,j}|n\in \mathbb{N},j \in \mathbb{M}2\}$, C1 and C2 are to limit the latency of each task to not exceed the requirements, C3  is to limit the user's overall latency does not exceed the requirements, C4 is to limit the processing power of the local device not to exceed its maximum processing power, C5 is to limit the transmission power of the local device not to exceed its maximum transmission power.

\subsection{Transmission Model}
{\color{black}{When the interference is not considered}}, the signal-to-noise ratio (SNR) of user n is
\begin{equation}\label{eqn07}
SNR_n=\frac{p_n^th_{n,m}^2}{\color{black}{B_w\mathcal{N}_0}}
\end{equation}
and then, the transmission rate (uplink) of user $n$ is calculated as:
\begin{equation}\label{eqn08}
r_n=Blog_2(1+ SNR_n).
\end{equation}
In \eqref{eqn07} and \eqref{eqn08}, {\color{black}{$B_w$}} represents the bandwidth of this channel,  $h_{n, m}$  represent the channel gain of user n   to MEC, and {\color{black}{$\mathcal{N}_0$ represents the noise spectral density}}, $p_n^t$ represents the transmission power of user $n$. Because the amount of data that needs to be offloaded is much larger than the amount of data that needs to be downloaded, {\color{black}{so the downlink is ignored in this paper}}\cite{7524497,7553459}.

\subsection{Computation Model}
Let $w_{n,j}$ {\color{black}{represent}} the CPU cycle required to calculate the unit (unit $j$ of user $n$), $d_{n,j}$ represents the computation input data (in bits) of the {\color{black}{ $j$ th unit}} of {\color{black}{$n$ th user of local devices}}.
\subsubsection*{Local Computing}
{\color{black}{when user chooses to process locally, use}} $f_{n,l}$ to represent the {\color{black}{CPU clock speed}} of user's device, so the latency in local for computing can be represented as:
\begin{equation}\label{eqn09}
t_{n,j}^l=\frac{w_{n,j}}{f_{n,l}}.
\end{equation}
where $ t_{n,j}^l$ {\color{black}{denotes}} the time required to complete unit $j$ of user $n$ in local device.

The energy required to complete unit $j$ in local device can be expressed by the following formula
\begin{equation}\label{eqn10}
E_{n,j}^l=\kappa w_{n,j}f^2_{n,l}
\end{equation}
In this case, $\kappa$ is the effective switched capacitance depending on the chip architecture\cite{miettinen2010energy}.
\subsubsection{MEC Computing}
when a user {\color{black}{decides}} to offload tasks to MEC, and {\color{black}{transmit}} a unit through a wireless network, the corresponding transmission latency and energy consumption will be generated. According to the communication model, the uplink transmission latency when user $n$ offload the task is:
\begin{equation}\label{eqn11}
t_{n,j}^t=\frac{d_{n,j}}{r_n},
\end{equation}
and the energy required to transmit unit $j$ can be expressed by the following formula:
\begin{equation}\label{eqn12}
E_{n,j}^{t}=P_{n,j}^{t}t_{n,j}^t.
\end{equation}
{\color{black}{where}} $ t_{n,j}^t$ represents the time required to transmit the {\color{black}{$j$ th unit of $n$ th user.}}
After the computing unit is offloaded to the MEC, the MEC will allocate certain computing resources to this unit, considering that the computing resources allocated by the MEC to each user are fixed. Let $f_{MEC}$ denote the computing resources allocated by the MEC, and the latency for the MEC to perform  {\color{black}{the $j$ th unit of $n$ th user}} is
\begin{equation}\label{eqn13}
t_{n,j}^m=\frac{w_{n,j}}{f_{MEC}}.
\end{equation}
Because MEC has a constant energy supply, {\color{black}{MEC's energy consumption need not be considered in this paper.}}

In this paper, {\color{black}{it is assumed}} that  MEC can only process {\color{black}{one}} unit at the same time, communication capacity can only support the transmission of a unit simultaneously, as shown in Fig. \ref{figure1} and Fig. \ref{figure2}. Blue {\color{black}{blocks}} represent transmission time, green {\color{black}{ones}} represent computing time, black ones {\color{black}{and yellow ones represent queuing for transmission, and queuing for computing respectively}}
At the beginning of this section, {\color{black}{it is introduced}} that each user generates multiple tasks and then they are split into multiple units, and needs to be processed. However, due to channel and processor limitations, these units cannot be processed at the same time. {\color{black}{There are}} two definitions of waiting time for the units which will be processed in local and waiting time for the units which will be processed in MEC. Suppose A is the set of units which will be processed in MEC, and A=\{1,2, ...,a\}, B is the set of units which will be processed in local device, and B=\{1,2, ...,b\},
\par \textbf{Definition 1}  (waiting time for the units which will be processed in local): The sum of all time, before the unit is processed by local device. {\color{black}{Use}} $WT_{n,k}^l$  to present the waiting time of unit $k$ of device $n$, which will be processed in local.
\par \textbf{Definition 2} (waiting time for the units which will be processed in MEC): The sum of all time, before the unit is processed in MEC. {\color{black}{Use}} $WT_{n,j}^m$  to present the waiting time of unit $j$ of device $n$, which will be processed in MEC.
\begin{figure}[t]
\centering
\includegraphics[width=8cm,height=3cm]{figure1.png}
\vspace{-1.5em}\caption{{\color{black}{Tasks which are processed in MEC}}}
\label{figure1}
\end{figure}

\begin{figure}[t]
\centering
\vspace{-1em}\includegraphics[width=8cm,height=3cm]{figure2.png}
\vspace{-1em}\caption{{\color{black}{Tasks which are processed in local}}}
\label{figure2}
\end{figure}
According to {\color{black}{the}} Definition 2, waiting time for the units which will be processed in MEC {\color{black}{is divided}} into {\color{black}{two}} parts, time of  waiting for transmission plus time of transmission ($WT3_{n,j}^m$), and
time before processing in MEC and after MEC receives the unit($WT4_{n,j}^m$), that means $ WT_{n,j}^m$ {\color{black}{can be divided}} into $WT3_{n,j}^m+ WT4_{n,j}^m$.

Assume that $unit_j\in A$\\
$when\ j=1$
\begin{equation}\label{eqn14}
WT3_{n,j}^m=t_{n,j}^t,\ \ \ \ \ WT4_{n,j}^m=0
\end{equation}
$when\ j >1$
{\color{black}{
\begin{equation}\label{eqn15}
\begin{split}
WT3_{n,j}^m&=WT3_{n,j-1}^m+t_{n,j}^t\\
WT4_{n,j}^m&=max(WT3_{n,j}^m, LT_{n,j-1}^m)-WT3_{n,j}^m
\end{split}
\end{equation}
In (\ref{eqn15}) , max means the maximum value between these two value.
\begin{equation}\label{eqn18}
\begin{split}
WT_{n,j}^m&=WT3_{n,j}^m+WT4_{n,j}^m\\
LT_{n,j}^m&=WT_{n,j}+t_{n,j}^m
\end{split}
\end{equation}
$LT_{n,j}^m$ means the latency of $unit_j$, $unit_j\in A$

Assume that $unit_k\in B$\\
so, in this case
\begin{equation}\label{eqn20}
WT_{n,k}^l=\sum_{c=1}^{c=j-1}t_{n,c}^l
\end{equation}
so that, the latency of unit $k$ can be expressed as ($unit_k\in B$):
\begin{equation}\label{eqn21}
LT_{n,k}^l=\sum_{c=1}^{c=k-1}t_{n,c}^l+t_{n,k}^l=\sum_{c=1}^{c=k}t_{n,c}^l
\end{equation}
So the latency of the whole system of user $n$ can be represent as:
\begin{equation}\label{eqn22}
Ts_n=max(LT_{n,b}^l, LT_{n,a}^m)
\end{equation}
Use $E_n^L$ to represent the total energy consumption of local device $n$:
\begin{equation}\label{eqn23}
E_n^L=\sum_{a=1}^{a=j} E^t_{n,a}+\sum_{b=1}^{b=k} E^l_{n,b}
\end{equation}
}}
\section{Proposed task offloading algorithm}
In this {\color{black}{section}}, first, several preprocessing methods {\color{black}{will be proposed}}, and perform some operations before task offloading, without affecting the reliability of the task, reduce the amount of task calculation and transmission, and then introduce the algorithm proposed for task offloading.
\subsection{Correlation in Time Domain of Tasks}
With the change of time, the task constantly updates its own information source to process the task. In the traditional task offloading, only consider how to change the task allocation mode, the processing frequency, transmission power to reduce latency and improve the energy-efficient, but in their algorithm, the data size does not change, which to a large extent, restricts the development of  task offloading, {\color{black}{this paper provide a new way}} to filter some unnecessary information to reduce the data size, so as to further reduce the latency and energy consumption. Correlation coefficient is a good way to be considered.

The correlation coefficient {\color{black} introduced in \cite{7000704}} is defined as:
\begin{equation}\label{eq1}
r(x,y) \triangleq  \frac{\mathsf{cov}(x,y)}{\sqrt{\mathsf{var}(x)\mathsf{var}(y)}}\ ,\ r(x,y) \in [-1,1].
\end{equation}
In this formula, x, y are the sources of information {\color{black}{which need to be compared}}. Suppose there is a task, denoted as $Task\ A$, {\color{black}{and}} denote the names of $Task\ A$ at different moments as $Task\ A1$, $Task\ A2$, $Task\ A3$, $Task\ A4$ in the order of time, $A1$ is the first and $A4$ is the last one, to reduce the number of processing, {\color{black}{correlation coefficient should be used}} between  them to decide whether to process these tasks, or process part of them. There are two ways to use the correlation coefficient.

The first way, set only one threshold to decide whether to process this task:  set a threshold $\alpha$, and then process the $Task\ A1$, and then calculate then correlation coefficient of $Task\ A1$ and $Task\ A2$. If the correlation coefficient between them is greater than $\alpha$, the $Task\ A2$ {\color{black}{will not be processed}}, and keep the $Task\ A1$ in memory, and the correlation coefficient between $Task\ A1$ and $Task\ A3$ {\color{black}{should be calculated}}, if it is greater than the  $\alpha$, $Task\ A3$ {\color{black}{will not be processed}} too, and continue to calculate the correlation coefficient between $Taks\ A1$ and $Task\ A4$, if not, $Task\ A3$ {\color{black}{will be processed}}, and keep $Task\ A3$ in memory, and then calculate the correlation coefficient between $Task\ A3$ and $Task\ A4$, and repeat with following Tasks.

The second way, set multi-threshold to decide how to process this task: {\color{black}{two thresholds}} as an example. First of all, set two thresholds, $\alpha$ and $\beta$, and $\alpha >\beta$, and  process the first task. When the correlation coefficient {\color{black}{between}} two adjacent tasks is greater than $\beta$ and smaller than $\alpha$, {\color{black}{just}} need to process the different part of this two tasks, when the correlation coefficient {\color{black}{between}} two tasks is smaller than $\beta$, {\color{black}{it can be considered}} the information of these two tasks are totally different, so, {\color{black}{the new task need to be processed}}, when the correlation coefficient {\color{black}{between}} two tasks is greater than $\alpha$,  {\color{black}{it can be considered}} the information of these two tasks are totally same, {\color{black}{the new task need not to be processed again}}
\subsection{Correlation in Task Domain of Tasks}
The correlation coefficient {\color{black}{ is considered in}} time domain, so as to reduce the amount of data, then the next step is to do the task splitting to get the units of these necessary task. {\color{black}{As said before}}, different tasks may split into the same unit, so the correlation of task domain came into being. How to use these correlation to further optimize{ \color{black}{the}} algorithm is the main content of this part, and {\color{black}{use}} $C_{o,p}$ to represent the correlation  between unit o and unit p. $C_{o,p}\in \{0, 0.5, 1\}$. when $C_{o,p}=0$,  that means that there is no relationship between unit o and unit p, both units need to be processed, when $C_{o,p}=1$,  that means unit o and unit p are total same, {\color{black}{and}} only need to process one of them and share the result, when $C_{o,p}=0.5$, that means  unit o and unit p are different units, but use the same source information.
\subsection{Units Allocation}
After {\color{black}{arranging units}} in the order required by the latency,  determine whether this unit needs to be assigned to the MEC for processing {\color{black}{from the first unit}}. At this time, the tree diagram{\color{black}{ needs to be drawn}}, as shown in Fig. \ref{figure42}, {\color{black}{use}}  four units as an example. Starting from the vertex first unit, there are two paths, one (unit will be processed in MEC Server) and zero (unit will be processed in local device). If the unit latency caused by the path meets the requirements of this unit, this path will be maintained and the next level of judgment will be made. If the latency does not meet the requirements, then drop this path. {\color{black}{Until finish}} the last task, and judgment whether the system latency is meet the requirement, if the answer is YES, keep this node, if not, drop this node, then all the feasible solutions {\color{black}{can be obtained}}. Then, for getting the optimal solution, some other optimization to transmission power and clock frequency {\color{black}{need to be done, and they will be introduced in the next few parts. }}
\begin{figure}[t]
\centering
\includegraphics[scale=0.27]{figure42.png}
\vspace{-1.5em}\caption{Binary Tree}
 \label{figure42}
\end{figure}
In the above situation, there is no correlation between units, {\color{black}{the next section will introduce}} the situation with there are correlation between units. when $C_{o,p}=1$,  the duplicate units {\color{black}{should be deleted}}, and then allocate the remaining units in the same way as said before. when $C_{o,p}=0.5$,  these units with a correlation degree of 0.5 {\color{black}{ need to be merged}}, that means merge them into a large task to carry out the process of task allocation(this allocation process is same as said before). If this large task is allocated to MEC server for processing, {\color{black}{it}} will reduce the latency and energy consumption caused by communication, that because they use the same source information. If this large task is allocated to local device for processing, the energy consumption and the latency will not change.
\subsection{Clock Frequency Allocation}
According to {\color{black}{(\ref{eqn10})}}, {\color{black}{if}} $w_{n,j}$ (the CPU cycle required to calculate the unit) not change, the smaller the value of $f_n^t$, the lower energy consumption will take.
\subsection{Transmission Power Allocation}
According to {\color{black}{(\ref{eqn07}),  (\ref{eqn08}), and (\ref{eqn12})}} and use $D$ to represent the total transmission volume of all units that need to be transmitted to MEC, so the energy consumed for transmission becomes E1
\begin{equation}
E1 ={\color{black}{\frac{D}{B_w}}\times log_{\eta}}2^{p^t_n}
\end{equation}
{\color{black}{$\eta=1+\frac{p_n^th_{n,m}^2}{{B_w}\mathcal{N}_0}$.}} So, the monotonic interval of E1 with respect to $p_n^t$, and take the  $p_n^t$value that minimizes E1 under conditions C1 to C5.
\subsection{Algorithm Flow}
Use the maximum $p_n^t$ of the local device and maximum $f_n^t$ of local device to find all combinations that satisfy the conditions of C1-C5, and record them as $\mathcal{V}$, and then use the algorithms proposed before find the smallest value of energy consumption that can be achieved after optimization in $\mathcal{V}$ , take the combination with the least energy consumption as {\color{black}{the}} final result. Therefore, the optimal clock frequency, transmission power, and unit allocation method can be obtained.

\section{Simulation results}
In this section, {\color{black} we evaluate our proposed task-offloading algorithm with three state-of-the-art baselines through MATLAB based simulations. It is considered in our simulation, that system contains four users and a single MEC. Each user has different number of tasks, and different type of task has different size.} Each user has {\color{black} its dedicated orthogonal channel to communicate to MEC}.

{\color{black}{The channel bandwidth is set }}to 20MHZ, and the channel conforms to the Rayleigh distribution. The SNR of the user terminal is set to 10dB, 20dB, 30dB, 40dB and 50dB in our simulations. The maximum computing rate in user's device is  $2\times10^9$cycles/s\cite{7914660}, and the maximum computing rate in MEC server is $20\times10^9$cycles/s \cite{7914660}, the size of each task is between 1-3M, and the number of processing revolutions they require is Between 1500-4500 cycles\cite{7914660}, and $\kappa =10^{-11}$ \cite{miettinen2010energy}. {\color{black}{There are}} two latency requirements, which are 50ms and 100ms.

\begin{figure}[t]
\centering
\vspace{-1.5em}	\subfigure[SNR=20,30dB]{
		\begin{minipage}[t]{1\linewidth}
			\centering
			\includegraphics[width=0.825\textwidth]{SNRM2030.png}
			\label{20}

		\end{minipage}%
	}%

	\quad
\subfigure[SNR=40,50dB]{
\begin{minipage}[t]{1\linewidth}
		\centering
              \includegraphics[width=0.825\textwidth]{SNRM4050.png}

			\label{40}
		\end{minipage}
	}%
	\centering
			\vspace{-1em}\caption{\color{black}{Energy consumption of five different algorithms }}
	\label{20304050}
\end{figure}

{\color{black}{In the experiment, {\color{black}{two proposed approaches simulated with the other three baselines}}.  Method 1 (Baseline 1): the most primitive task offloading algorithm, without considering any transmission power and other optimizations. Method 2 (Baseline 2): Further optimize the processing frequency and transmission power based on the Method 1\cite{7524497}.  Method 3 (Baseline 3): Based on Method 1 and Method 2, consider the impact of split tasks on task offloading\cite{8279411}. Method 4: Consider the relevance of task on time domain to reduce the amount of data. method 5: Base on Method 3, consider the correlation in time domain and in different tasks to reduce the amount of data.}}

The {\color{black}{Fig. \ref{20} Fig. \ref{40} show the energy consumption of the five different algorithms on the local device  when the SNR on the user terminal are 20dB, 30dB, 40dB and 50dB respectively. In the same SNR, our proposed algorithms have better performs
 Fig. \ref{SNRmethod} shows the probability of task failure under different methods and SNR.
From Fig. \ref{20304050}, and Fig. \ref{SNRmethod}, with the same SNR, our proposed algorithms have better performs in energy saving and reliability, and as the increase of SNR, with the same method, the energy cost may increase too, this is because we record that the energy consumption of the failed tasks that can be known in the stage of decision making is zero. The larger the SNR, the more tasks that can be processed, so it will also cause more energy consumption.}}

\section{Conclusion}
In this paper, it was studied how to improve the energy efficiency of local devices and reduce the latency in the process of task offloading. A novel task offloading algorithm has been proposed with the consideration of task correlation in both time domain and task domain to reduce the number of tasks to be transmitted. Moreover, a joint optimization has been studied for task allocation, transmission power and clock frequency. MATLAB based simulation results have demonstrated that the proposed task offloading algorithm can reduce the device energy consumption along various transmit power versus noise variance setup compare with the conventional one.

\begin{figure}[t]
\centering
\includegraphics[width=9.8cm,height=6.17 cm]{SNRmethod.png}
\vspace{-1.8em}\caption{{\color{black}{ Probability of task processing failure under different methods and SNR}}}
 \label{SNRmethod}
\end{figure}

{\color{black}{\section*{Acknowledgment}
The work was supported in part by European Com- mission under the framework of the Horizon2020 5G-Drive project, and in part by 5G Innovation Centre (5GIC) HEFEC grant.}}
\balance

\bibliographystyle{IEEEtran}