Self learning
202888986
ALBERT: A Lite BERT for Self supervised Learning of Language Representations
Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations and longer training times. To address these problems, we present two parameter reduction techniques to lower memory consumption and increase the training speed of BERT. Comprehensive empirical evidence shows that our proposed methods lead to models that scale much better compared to the original BERT. We also use a self supervised loss that focuses on modeling inter sentence coherence, and show it consistently helps downstream tasks with multi sentence inputs. As a result, our best model establishes new state of the art results on the GLUE, RACE, and \squad benchmarks while having fewer parameters compared to BERT large. The code and the pretrained models are available at this https URL.
ICLR
Zhenzhong Lan Mingda Chen Sebastian Goodman Kevin Gimpel Piyush Sharma Radu Soricut
203737303
SELF: Learning to Filter Noisy Labels with Self Ensembling
Deep neural networks (DNNs) have been shown to over fit a dataset when being trained with noisy labels for a long enough time. To overcome this problem, we present a simple and effective method self ensemble label filtering (SELF) to progressively filter out the wrong labels during training. Our method improves the task performance by gradually allowing supervision only from the potentially non noisy (clean) labels and stops learning on the filtered noisy labels. For the filtering, we form running averages of predictions over the entire training dataset using the network output at different training epochs. We show that these ensemble estimates yield more accurate identification of inconsistent predictions throughout training than the single estimates of the network at the most recent training epoch. While filtered samples are removed entirely from the supervised training loss, we dynamically leverage them via semi supervised learning in the unsupervised loss. We demonstrate the positive effect of such an approach on various image classification tasks under both symmetric and asymmetric label noise and at different noise ratios. It substantially outperforms all previous works on noise aware learning across different datasets and can be applied to a broad set of network architectures.
ICLR
Duc Tam Nguyen Chaithanya Kumar Mummadi Thi-Phuong-Nhung Ngo Thi Hoai Phuong Nguyen Laura Beggel Thomas Brox
21728524
A robust self learning method for fully unsupervised cross lingual mappings of word embeddings
Recent work has managed to learn cross lingual word embeddings without parallel data by mapping monolingual embeddings to a shared space through adversarial training. However, their evaluation has focused on favorable conditions, using comparable corpora or closely related languages, and we show that they often fail in more realistic scenarios. This work proposes an alternative approach based on a fully unsupervised initialization that explicitly exploits the structural similarity of the embeddings, and a robust self learning algorithm that iteratively improves this solution. Our method succeeds in all tested scenarios and obtains the best published results in standard datasets, even surpassing previous supervised systems. Our implementation is released as an open source project at https:/github.com/artetxem/vecmap.
ACL
Mikel Artetxe Gorka Labaka Eneko Agirre
202148356
Binary differential evolution with self learning for multi objective feature selection
Abstract Feature selection is an important data preprocessing method. This paper studies a new multi objective feature selection approach, called the Binary Differential Evolution with self learning (MOFS BDE) Three new operators are proposed and embedded into the MOFS BDE to improve its performance. The novel binary mutation operator based on probability difference can guide individuals to rapidly locate potentially optimal areas, the developed One bit Purifying Search operator (OPS) can improve the self learning capability of the elite individuals located in the optimal areas, and the efficient non dominated sorting operator with crowding distance can reduce the computational complexity of the selection operator in the differential evolution. Experimental results on a series of public datasets show that the effective combination of the binary mutation and OPS makes our MOFS BDE achieve a trade off between local exploitation and global exploration. The proposed method is competitive in comparison with some representative genetic algorithm particle swarm differential evolution and artificial bee colony based feature selection algorithms.
Inf. Sci.
Yanweng Zhang Dunwei Gong Xiao-Zhi Gao Tian Tian Xiaoyan Sun
219687798
Bootstrap Your Own Latent: A New Approach to Self Supervised Learning
We introduce Bootstrap Your Own Latent (BYOL) a new approach to self supervised image representation learning. BYOL relies on two neural networks, referred to as online and target networks, that interact and learn from each other. From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view. At the same time, we update the target network with a slow moving average of the online network. While state of the art methods rely on negative pairs, BYOL achieves a new state of the art without them. BYOL reaches $74.3\ top 1 classification accuracy on ImageNet using a linear evaluation with a ResNet 50 architecture and $79.6\ with a larger ResNet. We show that BYOL performs on par or better than the current state of the art on both transfer and semi supervised benchmarks. Our implementation and pretrained models are given on GitHub.
NeurIPS
Jean-Bastien Grill Florian Strub Florent Altch'e Corentin Tallec Pierre H Richemond Elena Buchatskaya Carl Doersch Bernardo Avila Pires Zhaohan Daniel Guo Mohammad Gheshlaghi Azar Bilal Piot Koray Kavukcuoglu Remi Munos Michal Valko
56482410
DIoT: A Federated Self learning Anomaly Detection System for IoT
IoT devices are increasingly deployed in daily life. Many of these devices are, however, vulnerable due to insecure design, implementation, and configuration. As a result, many networks already have vulnerable IoT devices that are easy to compromise. This has led to a new category of malware specifically targeting IoT devices. However, existing intrusion detection techniques are not effective in detecting compromised IoT devices given the massive scale of the problem in terms of the number of different types of devices and manufacturers involved. In this paper, we present DIoT, an autonomous self learning distributed system for detecting compromised IoT devices. DIoT builds effectively on device type specific communication profiles without human intervention nor labeled data that are subsequently used to detect anomalous deviations in devices' communication behavior, potentially caused by malicious adversaries. DIoT utilizes a federated learning approach for aggregating behavior profiles efficiently. To the best of our knowledge, it is the first system to employ a federated learning approach to anomaly detection based intrusion detection. Consequently, DIoT can cope with emerging new and unknown attacks. We systematically and extensively evaluated more than 30 off the shelf IoT devices over a long term and show that DIoT is highly effective (95.6% detection rate) and fast (257 ms) at detecting devices compromised by, for instance, the infamous Mirai malware. DIoT reported no false alarms when evaluated in a real world smart home deployment setting.
2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS)
Thien Duc Nguyen Samuel Marchal Markus Miettinen Hossein Fereidooni N Asokan Ahmad-Reza Sadeghi
199453105
Deep Self Learning From Noisy Labels
ConvNets achieve good results when training from clean data, but learning from noisy labels significantly degrades performances and remains challenging. Unlike previous works constrained by many conditions, making them infeasible to real noisy cases, this work presents a novel deep self learning framework to train a robust network on the real noisy datasets without extra supervision. The proposed approach has several appealing benefits. (1) Different from most existing work, it does not rely on any assumption on the distribution of the noisy labels, making it robust to real noises. (2) It does not need extra clean supervision or accessorial network to help training. (3) A self learning framework is proposed to train the network in an iterative end to end manner, which is effective and efficient. Extensive experiments in challenging benchmarks such as Clothing1M and Food101 N show that our approach outperforms its counterparts in all empirical settings.
2019 IEEE/CVF International Conference on Computer Vision (ICCV)
Jiangfan Han Ping Luo Xiaogang Wang
208617491
Self Supervised Learning of Pretext Invariant Representations
The goal of self supervised learning from images is to construct image representations that are semantically meaningful via pretext tasks that do not require semantic annotations. Many pretext tasks lead to representations that are covariant with image transformations. We argue that, instead, semantic representations ought to be invariant under such transformations. Specifically, we develop Pretext Invariant Representation Learning (PIRL, pronounced as `pearl' that learns invariant representations based on pretext tasks. We use PIRL with a commonly used pretext task that involves solving jigsaw puzzles. We find that PIRL substantially improves the semantic quality of the learned image representations. Our approach sets a new state of the art in self supervised learning from images on several popular benchmarks for self supervised learning. Despite being unsupervised, PIRL outperforms supervised pre training in learning image representations for object detection. Altogether, our results demonstrate the potential of self supervised representations with good invariance properties.
2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Ishan Misra Laurens van der Maaten
219787939
Online self learning
In this paper four short sessions are reviewed in the theme of introductory programming. The sessions were performed in four different student groups from the age twelve to seventeen in two different schools. The paper summarizes important input and output properties of the groups, like the preliminary studies of Informatics and the grades in both Informatics and Mathematics if available also the results of the learning process during the sessions. The focus is on the skills of programming and self learning, and the differences of it in respect to the age, the gender and the preliminary studies.
Eva Adamko
15780954
End to End Learning for Self Driving Cars
We trained a convolutional neural network (CNN) to map raw pixels from a single front facing camera directly to steering commands. This end to end approach proved surprisingly powerful. With minimum training data from humans the system learns to drive in traffic on local roads with or without lane markings and on highways. It also operates in areas with unclear visual guidance such as in parking lots and on unpaved roads. The system automatically learns internal representations of the necessary processing steps such as detecting useful road features with only the human steering angle as the training signal. We never explicitly trained it to detect, for example, the outline of roads. Compared to explicit decomposition of the problem, such as lane marking detection, path planning, and control, our end to end system optimizes all processing steps simultaneously. We argue that this will eventually lead to better performance and smaller systems. Better performance will result because the internal components self optimize to maximize overall system performance, instead of optimizing human selected intermediate criteria, e.g. lane detection. Such criteria understandably are selected for ease of human interpretation which doesn't automatically guarantee maximum system performance. Smaller networks are possible because the system learns to solve the problem with the minimal number of processing steps. We used an NVIDIA DevBox and Torch 7 for training and an NVIDIA DRIVE(TM) PX self driving car computer also running Torch 7 for determining where to drive. The system operates at 30 frames per second (FPS)
ArXiv
Mariusz Bojarski David W del Testa Daniel Dworakowski Bernhard Firner Beat Flepp Prasoon Goyal Lawrence D Jackel Mathew Monfort Urs Muller Jiakai Zhang X Zhang Jake Zhao Karol Zieba