attention image
4717445
Learn To Pay Attention
We propose an end to end trainable attention module for convolutional neural network (CNN) architectures built for image classification. The module takes as input the 2D feature vector maps which form the intermediate representations of the input image at different stages in the CNN pipeline, and outputs a 2D matrix of scores for each map. Standard CNN architectures are modified through the incorporation of this module, and trained under the constraint that a convex combination of the intermediate 2D feature vectors, as parameterised by the score matrices, must \textit{alone} be used for classification. Incentivised to amplify the relevant and suppress the irrelevant or misleading, the scores thus assume the role of attention values. Our experimental observations provide clear evidence to this effect: the learned attention maps neatly highlight the regions of interest while suppressing background clutter. Consequently, the proposed function is able to bootstrap standard CNN architectures for the task of image classification, demonstrating superior generalisation over 6 unseen benchmark datasets. When binarised, our attention maps outperform other CNN based attention maps, traditional saliency maps, and top object proposals for weakly supervised segmentation as demonstrated on the Object Discovery dataset. We also demonstrate improved robustness against the fast gradient sign method of adversarial attack.
ICLR
Saumya Jetley Nicholas A Lord Namhoon Lee Philip H S Torr
53106274
A2 Nets: Double Attention Networks
Learning to capture long range relations is fundamental to image/video recognition. Existing CNN models generally rely on increasing depth to model such relations which is highly inefficient. In this work, we propose the "double attention block" a novel component that aggregates and propagates informative global features from the entire spatio temporal space of input images/videos, enabling subsequent convolution layers to access features from the entire space efficiently. The component is designed with a double attention mechanism in two steps, where the first step gathers features from the entire space into a compact set through second order attention pooling and the second step adaptively selects and distributes features to each location via another attention. The proposed double attention block is easy to adopt and can be plugged into existing deep neural networks conveniently. We conduct extensive ablation studies and experiments on both image and video recognition tasks for evaluating its performance. On the image recognition task, a ResNet 50 equipped with our double attention blocks outperforms a much larger ResNet 152 architecture on ImageNet 1k dataset with over 40% less the number of parameters and less FLOPs. On the action recognition task, our proposed model achieves the state of the art results on the Kinetics and UCF 101 datasets with significantly higher efficiency than recent works.
NeurIPS
Yunpeng Chen Yannis Kalantidis Jianshu Li Shuicheng Yan Jiashi Feng
2148670
Attention Based Models for Text Dependent Speaker Verification
Attention based models have recently shown great performance on a range of tasks, such as speech recognition, machine translation, and image captioning due to their ability to summarize relevant information that expands through the entire length of an input sequence. In this paper, we analyze the usage of attention mechanisms to the problem of sequence summarization in our end to end text dependent speaker recognition system. We explore different topologies and their variants of the attention layer, and compare different pooling methods on the attention weights. Ultimately, we show that attention based models can improves the Equal Error Rate (EER) of our speaker verification system by relatively 14% compared to our non attention LSTM baseline model.
2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
F R Chowdhury Quan Wang Ignacio Lopez-Moreno Li Wan
4625261
Improved Fusion of Visual and Language Representations by Dense Symmetric Co attention for Visual Question Answering
A key solution to visual question answering (VQA) exists in how to fuse visual and language features extracted from an input image and question. We show that an attention mechanism that enables dense, bi directional interactions between the two modalities contributes to boost accuracy of prediction of answers. Specifically, we present a simple architecture that is fully symmetric between visual and language representations, in which each question word attends on image regions and each image region attends on question words. It can be stacked to form a hierarchy for multi step interactions between an image question pair. We show through experiments that the proposed architecture achieves a new state of the art on VQA and VQA 2.0 despite its small size. We also present qualitative evaluation, demonstrating how the proposed attention mechanism can generate reasonable attention maps on images and questions, which leads to the correct answer prediction.
2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
Duy-Kien Nguyen Takayuki Okatani
945386
Dual Attention Networks for Multimodal Reasoning and Matching
We propose Dual Attention Networks (DANs) which jointly leverage visual and textual attention mechanisms to capture fine grained interplay between vision and language. DANs attend to specific regions in images and words in text through multiple steps and gather essential information from both modalities. Based on this framework, we introduce two types of DANs for multimodal reasoning and matching, respectively. The reasoning model allows visual and textual attentions to steer each other during collaborative inference, which is useful for tasks such as Visual Question Answering (VQA) In addition, the matching model exploits the two attention mechanisms to estimate the similarity between images and sentences by focusing on their shared semantics. Our extensive experiments validate the effectiveness of DANs in combining vision and language, achieving the state of the art performance on public benchmarks for VQA and image text matching.
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Hyeonseob Nam Jung-Woo Ha Jeonghee Kim
20970043
Attentive Collaborative Filtering: Multimedia Recommendation with Item and Component Level Attention
Multimedia content is dominating today's Web information. The nature of multimedia user item interactions is 1/0 binary implicit feedback (e.g. photo likes, video views, song downloads, etc. which can be collected at a larger scale with a much lower cost than explicit feedback (e.g. product ratings) However, the majority of existing collaborative filtering (CF) systems are not well designed for multimedia recommendation, since they ignore the implicitness in users' interactions with multimedia content. We argue that, in multimedia recommendation, there exists item and component level implicitness which blurs the underlying users' preferences. The item level implicitness means that users' preferences on items (e.g. photos, videos, songs, etc. are unknown, while the component level implicitness means that inside each item users' preferences on different components (e.g. regions in an image, frames of a video, etc. are unknown. For example, a 'view' on a video does not provide any specific information about how the user likes the video (i.e.item level) and which parts of the video the user is interested in (i.e.component level) In this paper, we introduce a novel attention mechanism in CF to address the challenging item and component level implicit feedback in multimedia recommendation, dubbed Attentive Collaborative Filtering (ACF) Specifically, our attention model is a neural network that consists of two attention modules: the component level attention module, starting from any content feature extraction network (e.g. CNN for images/videos) which learns to select informative components of multimedia items, and the item level attention module, which learns to score the item preferences. ACF can be seamlessly incorporated into classic CF models with implicit feedback, such as BPR and SVD+ and efficiently trained using SGD. Through extensive experiments on two real world multimedia Web services: Vine and Pinterest, we show that ACF significantly outperforms state of the art CF methods.
SIGIR
Jingyuan Chen Hanwang Zhang Xiangnan He Liqiang Nie Wenjin Liu Tat-Seng Chua
149721293
Attention image data for objective measurement of video quality loss
Bozidar Janko J W F Edwards
1921173
Attention Based Models for Speech Recognition
Recurrent sequence generators conditioned on input data through an attention mechanism have recently shown very good performance on a range of tasks including machine translation, handwriting synthesis [1,2] and image caption generation [3] We extend the attention mechanism with features needed for speech recognition. We show that while an adaptation of the model used for machine translation in [2] reaches a competitive 18.7% phoneme error rate (PER) on the TIMET phoneme recognition task, it can only be applied to utterances which are roughly as long as the ones it was trained on. We offer a qualitative explanation of this failure and propose a novel and generic method of adding location awareness to the attention mechanism to alleviate this issue. The new method yields a model that is robust to long inputs and achieves 18% PER in single utterances and 20% in 10 times longer (repeated) utterances. Finally, we propose a change to the attention mechanism that prevents it from concentrating too much on single frames, which further reduces PER to 17.6% level.
NIPS
Jan Chorowski Dzmitry Bahdanau Dmitriy Serdyuk Kyunghyun Cho Yoshua Bengio
4570002
Attention based Ensemble for Deep Metric Learning
Deep metric learning aims to learn an embedding function, modeled as deep neural network. This embedding function usually puts semantically similar images close while dissimilar images far from each other in the learned embedding space. Recently, ensemble has been applied to deep metric learning to yield state of the art results. As one important aspect of ensemble, the learners should be diverse in their feature embeddings. To this end, we propose an attention based ensemble, which uses multiple attention masks, so that each learner can attend to different parts of the object. We also propose a divergence loss, which encourages diversity among the learners. The proposed method is applied to the standard benchmarks of deep metric learning and experimental results show that it outperforms the state of the art methods by a significant margin on image retrieval tasks.
ECCV
Wonsik Kim Bhavya Goyal Kunal Chawla Jungmin Lee Keunjoo Kwon
195654199
Depth Attentional Features for Single Image Rain Removal
Rain is a common weather phenomenon, where object visibility varies with depth from the camera and objects faraway are visually blocked more by fog than by rain streaks. Existing methods and datasets for rain removal, however, ignore these physical properties, thereby limiting the rain removal efficiency on real photos. In this work, we first analyze the visual effects of rain subject to scene depth and formulate a rain imaging model collectively with rain streaks and fog; by then, we prepare a new dataset called RainCityscapes with rain streaks and fog on real outdoor photos. Furthermore, we design an end to end deep neural network, where we train it to learn depth attentional features via a depth guided attention mechanism, and regress a residual map to produce the rain free image output. We performed various experiments to visually and quantitatively compare our method with several state of the art methods to demonstrate its superiority over the others.
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Xiaowei Hu Chi-Wing Fu Lei Zhu Pheng-Ann Heng