self-attention
46898260
Self Attention Generative Adversarial Networks
In this paper, we propose the Self Attention Generative Adversarial Network (SAGAN) which allows attention driven, long range dependency modeling for image generation tasks. Traditional convolutional GANs generate high resolution details as a function of only spatially local points in lower resolution feature maps. In SAGAN, details can be generated using cues from all feature locations. Moreover, the discriminator can check that highly detailed features in distant portions of the image are consistent with each other. Furthermore, recent work has shown that generator conditioning affects GAN performance. Leveraging this insight, we apply spectral normalization to the GAN generator and find that this improves training dynamics. The proposed SAGAN achieves the state of the art results, boosting the best published Inception score from 36.8 to 52.52 and reducing Frechet Inception distance from 27.62 to 18.65 on the challenging ImageNet dataset. Visualization of the attention layers shows that the generator leverages neighborhoods that correspond to object shapes rather than local regions of fixed shape.
ICML
Han Zhang Ian J Goodfellow Dimitris N Metaxas Augustus Odena
10287790
Self focused attention and negative affect: a meta analysis.
This meta analysis synthesized 226 effect sizes reflecting the relation between self focused attention and negative affect (depression, anxiety, negative mood) The results demonstrate the multifaceted nature of self focused attention and elucidate major controversies in the field. Overall, self focus was associated with negative affect. Several moderators qualified this relationship. Self focus and negative affect were more strongly related in clinical and female dominated samples. Rumination yielded stronger effect sizes than nonruminative self focus. Self focus on positive self aspects and following a positive event were related to lower negative affect. Most important, an interaction between foci of self attention and form of negative affect was found: Private self focus was more strongly associated with depression and generalized anxiety, whereas public self focus was more strongly associated with social anxiety.
Psychological bulletin
Nilly Mor Jennifer R Winquist
15432031
Attention, self regulation and consciousness.
Consciousness has many aspects. These include awareness of the world, feelings of control over one's behaviour and mental state (volition) and the notion of continuing self. Focal (executive) attention is used to control details of our awareness and is thus closely related to volition. Experiments suggest an integrated network of neural areas involved in executive attention. This network is associated with our voluntary ability to select among competing items, to correct error and to regulate our emotions. Recent neuroimaging studies suggest that these various functions involve separate areas of the anterior cingulate. We have adopted a strategy of using marker tasks, shown to activate the brain area by imaging studies, as a means of tracing the development of attentional networks. Executive attention appears to develop first to regulate distress during the first year of life. During later childhood the ability to regulate conflict among competing stimuli builds upon the earlier cingulate anatomy to provide a means of cognitive control. During childhood the activation of cingulate structures relates both to the child's success on laboratory tasks involving conflict and to parental reports of self regulation and emotional control. These studies indicate a start in understanding the anatomy, circuitry and development of executive attention networks that serve to regulate both cognition and emotion.
Philosophical transactions of the Royal Society of London. Series B, Biological sciences
Michael I Posner Mary K Rothbart
7170900
Short term meditation training improves attention and self regulation
Recent studies suggest that months to years of intensive and systematic meditation training can improve attention. However, the lengthy training required has made it difficult to use random assignment of participants to conditions to confirm these findings. This article shows that a group randomly assigned to 5 days of meditation practice with the integrative body mind training method shows significantly better attention and control of stress than a similarly chosen control group given relaxation training. The training method comes from traditional Chinese medicine and incorporates aspects of other meditation and mindfulness training. Compared with the control group, the experimental group of 40 undergraduate Chinese students given 5 days of 20 min integrative training showed greater improvement in conflict scores on the Attention Network Test, lower anxiety, depression, anger, and fatigue, and higher vigor on the Profile of Mood States scale, a significant decrease in stress related cortisol, and an increase in immunoreactivity. These results provide a convenient method for studying the influence of meditation training by using experimental and control methods similar to those used to test drugs or other interventions.
Proceedings of the National Academy of Sciences
Yi-Yuan Tang Yinghua Ma Junhong Wang Yaxin Fan Shigang Feng Qilin Lu Qingbao Yu Danni Sui Mary K Rothbart Ming Fan Michael I Posner
20894553
Attention processes in the maintenance and treatment of social phobia: hypervigilance, avoidance and self focused attention.
Four domains of evidence regarding the relationship between attention and social phobia are reviewed: (1) possible maintaining factor, (2) causal relationship, (3) specific relationship and (4) mediator of change during treatment. Two areas of research are covered and integrated: vigilance avoidance of social threat stimuli and self focused attention. There is empirical support for these processes as possible maintaining factors that are specific to patients and non clinical samples with high levels of social anxiety. There is reasonable evidence to promote the use of attentional strategies in the treatment of social phobia, although better controlled studies are required. We propose six overlapping mechanisms that could be responsible for change: reduced vigilance; reduced avoidance; reduced self focused attention; mindfulness; increased attentional control; increased self esteem.
Clinical psychology review
Susan M Bogels Warren Mansell
142694007
Attention and Self Regulation: A Control Theory Approach to Human Behavior
I. Background. 1. Introduction. A Statement of Intentions. Organizational Plan. 2. Cybernetics, Information, and Control. Information. Levels of Information. Classification of Input. Control. Negative Feedback Loop and TOTE Unit. Organizations Among Feedback Loops. Positive Feedback Loops. Hierarchical Organization and Time Scales. Section Summary. Applications to Living Systems. Homeostasis. Adaptation Level Theory. Behavior Regulation. Information, Control, and Personality Social Psychology. 3. Focus of Attention, Inside and Outside the Laboratory. Defining Terms. Attention. Self Focus and Environment Focus. Maintaining the Distinction. Varying Attention in Research. General Issues in Research Strategy. Experimental Manipulations and Individual Differences. Experimental Manipulations: Their Nature and Timing. Manipulations of Self Attention. Validation. Individual Differences in Self Focus. Validation. Convergence of Manipulation and Disposition. Validity. Private and Public Self Awareness. Beyond the Laboratory. Naturalistic Variations in Attentional Focus. Attention to the Environment. Attention to the Self. Conclusion. II. Information and the Use of Recognitory Schemas. 4. Cognitive Theory: Schemas, Attributes, and Decision Biases. Models of Abstraction. Instance Theories. Prototype Theories. Frequency Distribution Theories. Section Summary. Schemas and Prototypes. Abstraction: Some Additional Issues. A Hierarchy of Attribute Qualities. Frames and Scripts. Decision Making. Biases in Decision Making. Section Summary. 5. Focus on the Environment: Perception of Places and Persons. Behavioral Settings and Environmental Taxonomies. Person Perception. Traits as Recognitory Schemas. Determinants of Initial Category Placement. Consequences of Initial Category Placement. Organization of Perception of Sequences. Segmenting of Behavior Units. Focus of Attention and Perception of the Environment. 6. Focus on the Self: Perception of Self Aspects. Trait Schemas and the Self. Self Schemas and Encoding. Component Schemas and Degrees of Schematicity. Conclusion. Access to Self Schemas. Self Focus and Encoding by Self Reference. Self Focus and Activation of the Self Schema. Access and Attribution. Internal States: Emotions and Symptoms. Three Sources of Influence on Perceived Internal States. Impact of Each Element on Subjective Experience. Schemas and Symptom Distress. Evoked Schemas and the By Passing of Awareness. Summary. III. Attention and Motivation. 7. Standards of Behavior. Standards. What is a Behavioral Standard? Categorization and the Specification of Behavior. Section Summary. A Hierarchy of Standards. Levels of Behavioral Standards. Conceptual Levels and the Physical. Execution of Behavior. Distinctions Within Levels. Summary. Programs, Principles, and Consciousness. Effects of Attending to Well Learned Behavior. Closing Comment. 8. Self Focus and Feedback Loops. Discrepancy Reduction. Theory. Duval and Wicklund's Theory. The Comparison of Present State Versus Standard. Self Awareness and Information Seeking. Self Consciousness and Information Seeking. Self Consciousness and the Seeking of Diagnostic Information. Self Awareness and the Seeking of Diagnostic Information. Conclusions and Boundary Condition. Behavioral Matching to Standard. Nonprovoked Aggression. Responses to Erotica. Children's Use of Standards. Section Summary. Discrepancy Enlargement. Reactance. Self Awareness and Reactance. Self Consciousness and Reactance. Self Attention, Reactance, and Feedback Loops. Negative Reference Groups. Self Consciousness and Use of Negative Reference Groups. Positive Feedback Loop. Social Comparison: A Theoretical Integration. 9. Absence of Regulation, and Misregulation. A Conceptual Distinction. The Absence of Regulation. Causes. Deindividuation and the Absence of Regulation. Remaining Issues Regarding Deindividuation Related Issues. Misregulation. Ilustrations from Behavioral Medicine. Illustrations from. Social Behavior. Summary. IV. Interruption, Expectancy and the Reassertion Withdrawal Decision. 10. Theory: Interrupting the Feedback Loop, and the Role of Expectancy. Interruption of the System: An Example. Response to Interruption: Assessment of Outcome Expectancy. Bases of Expectancies. Prior Success and Failure. Other Bases for Expectancies. Consequences of Expectancy Judgments. Behavioral Consequences. Relationship to Other Theories. Affective Consequences of Expectancy Judgments. Summary. 11. Research: Persistence and Task Performance. Early Research. Discrepancy and the Avoidance of Self Focusing Stimul. Discrepancy and Selective Exposure to Self. Subsequent Research Concerning Outcome Expectancy. Avoidance and the Flexibility of the Discrepancy. Self Focus and Expectancies: Persistence and Withdrawal. Self Consciousness, Prior Outcomes, and Persistence. Self Esteem, Self Attention, and Task Performance. Self Awareness, Relevance, and Persistence. Outcome Expectancy and Efficacy Expectancy. Links Between Cognition and Emotion in Achievement. Attributions, Expectancy, and Achievement Behavior. Prior Outcomes, Attribution, and Maze Performance. Summary. 12. Research: Anxiety Related Behavior. Self Attention and Fear. Self Awareness and Phobic Behavior. Self Consciousness, Standards, and Fear. Expectancies, Fear, and the Approach Withdrawal Decision. Self Attention and Test Anxiety. Test Anxiety, Self Focus, and Facilitation of Performance. Theoretical Comparisons. Theories of Test Anxiety. Self Efficacy. Summary. 13. Additional Conceptual Issues: Achievement Motivation, Helplessness, and Egotism. Achievement Motivation. Atkinson's Formulation. Relationship to Present Model. The Role of Causal Attributions. Section Summary. Helplessness. Reassertion and Giving Up. Attributions, Behavior, and Affect. Section Summary. Egotism and Self Esteem Maintenance. Control Theory and Self Esteem Maintenance. Egotism and Attribution. Egotism and Helplessness. Expectancies, Outcomes, and the Effect of Perceived Difficulty. Conceptual Similarities Self Esteem Protection: Public or Private? Section Summary. Summary. V. Implications for Specific Problems in Social and Personality Psychology. 14. Relationship Between Self Report and Behavior. Problems of Measurement. Levels of Specificity of Self Report and Behavior. Insufficient Sampling of Behavior. Assessment of Behavioral Inconsistency. Conceptual Problems. Veridicality of Self Reports. The Role of Direct Experience in Attitude Formation. Relevance of Attitudes as Behavioral Standards. Self Awareness and the Use of Evoked Attitudes. Types of Self Report. 15. Social Facilitation. Drive Theory and Social Facilitation. Alternative Drive Theories. Control Theory and Social Facilitation. Comparisons Among Theories. Evidence Regarding Mediating States. Self Focus and Comparison with Standards. Drive. Palmar Sweat and Social Facilitation. Results and Discussion. Additional Questions. Social Interference. Comparisons Among Manipulations. Physiological Response Patterning. 16. Private and Public Selves. Self Completion, or Self Presentation? Impact on Research Areas. Self, or Selves? Aspects of Self. Implications. The Consciousness of Public Versus Private Self Aspects. Public and Private Selves in a Compliance Setting. Public and Private Selves and Reactions to Coercion. The Public Expression of Privately Held Opinions. Private and Public Self Awareness Reinterpreting Other Research. Section Summary. Self Monitoring. Summary, Critique, and Integration. 17. Cognitive Dissonance. Self Attention and Dissonance. Public and Private Self Aspects Tests of the Public Private Analysis. Self Focus Manipulations and Dissonance Reduction. Self Consciousness and Dissonance Reduction. Relationship to Other Dissonance Research. Dissonance, or Impression Management? VI. Conclusion. 18. Afterword: Theory and Meta Theory. Changes in the Learning Paradigm. Meta Theoretical Issues. Reinforcement. Control Theory. Conclusion. References. Author Index.
Charles S Carver Michael F Scheier
145299940
Self Consciousness, Self Attention, and Social Interaction
Two experiments were conducted to test the effects of self focused attention on positive and negative social interactions. In the first study, the behavior of dispositionally high and low publicly self conscious women was examined in an interpersonal situation involving rejection by a group. It was hypothesized that persons high in self consciou sness, being more aware of how they are perceived by others, would be more sensitive and react more negatively to the rejection than those low in self consciou sness. The predictions were confirmed. In Experiment 2, female subjects were presented with favorable or unfavorable feedback in the context of an interview, and self attention was experimentally manipulated by exposing half the subjects to their images in a mirror. Self awareness increased the negative response to the negative evaluation and tended to increase the positivity of the positive evaluation. The implications of selfawareness theory for the social self and social interaction are discussed. Everyday observations as well as theoretical approaches to social behavior suggest that in the presence of others, one is apt to become self conscious, that is, aware of the self as a social object that can be observed and evaluated by others. Goffman (1959) has argued quite persuasively that when one is attending to and involved in an ongoing interaction,
Allan Fenigstein
3292002
Graph Attention Networks
We present graph attention networks (GATs) novel neural network architectures that operate on graph structured data, leveraging masked self attentional layers to address the shortcomings of prior methods based on graph convolutions or their approximations. By stacking layers in which nodes are able to attend over their neighborhoods' features, we enable (implicitly) specifying different weights to different nodes in a neighborhood, without requiring any kind of costly matrix operation (such as inversion) or depending on knowing the graph structure upfront. In this way, we address several key challenges of spectral based graph neural networks simultaneously, and make our model readily applicable to inductive as well as transductive problems. Our GAT models have achieved or matched state of the art results across four established transductive and inductive graph benchmarks: the Cora, Citeseer and Pubmed citation network datasets, as well as a protein protein interaction dataset (wherein test graphs remain unseen during training)
ICLR
Petar Velickovic Guillem Cucurull Arantxa Casanova Adriana Romero Pietro Lio' Yoshua Bengio
13756489
Attention is All you Need
The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English to German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English to French translation task, our model establishes a new single model state of the art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.
NIPS
Ashish Vaswani Noam M Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Lukasz Kaiser Illia Polosukhin
1862853
Self focused attention in clinical disorders: review and a conceptual model.
Working largely independently, numerous investigators have explored the role of self focused attention in various clinical disorders. This article reviews research examining increased self focused attention in these disorders. Results indicate that regardless of the particular disorder under investigation, a heightened degree of self focused attention is found. Hence, as ordinarily conceptualized, self focused attention has little discriminatory power among different psychological disorders. Using information processing constructs, a somewhat different model of self focused attention is proposed, and it is suggested that certain deviations in this process constitute a psychopathological kind of attention. A meta construct model of descriptive psychopathology is then outlined to examine how certain aspects of attention can be considered specific to certain disorders and others common to different disorders.
Psychological bulletin
Rick E Ingram