Deep Transfer Learning: A new deep learning glitch
classification method for advanced LIGO

arXiv:1706.07446v1 [gr-qc] 22 Jun 2017

Daniel George1,2 , Hongyu Shen1,3 and E. A. Huerta1
1

NCSA, University of Illinois at Urbana-Champaign, Urbana, Illinois, 61801
Department of Astronomy, University of Illinois at Urbana-Champaign, Urbana,
Illinois, 61801
3
Department of Statistics, University of Illinois at Urbana-Champaign, Urbana,
Illinois, 61801
2

26 June 2017
Abstract.
The exquisite sensitivity of the advanced LIGO detectors has enabled the
detection of multiple gravitational wave signals, thereby establishing gravitational wave
astrophysics as an active field of research. The sophisticated design of these detectors
mitigates the effect of most types of noise. However, advanced LIGO data streams
are contaminated by numerous artifacts known as glitches — non-Gaussian noise
transients with complex morphologies. Given their high rate of occurrence, glitches
can lead to false coincident detections, obscure and even mimic true gravitational wave
signals. Therefore, successfully characterizing and removing glitches from advanced
LIGO data is of utmost importance. In this article, we present the first application
of Deep Transfer Learning for glitch classification, showing that knowledge from
deep learning algorithms trained for real-world object recognition can be transferred
for classifying glitches in time-series based on their spectrogram images. Using the
Gravity Spy dataset, containing hand-labeled, multi-duration spectrograms obtained
from real LIGO data, we demonstrate that this method enables optimal use of very
deep convolutional neural networks for classification given small training datasets,
significantly reduces the time for training the networks, and achieves state-of-the-art
accuracy above 98.8%, with perfect precision-recall on 8 out of 22 classes. Furthermore,
new types of glitches can be classified accurately given few labeled examples with this
technique. Once trained via transfer learning, we show that the convolutional neural
networks can be truncated and used as excellent feature extractors for unsupervised
clustering methods to identify new classes based on their morphology, without any
labeled examples. Therefore, this provides a new framework for dynamic glitch
classification for gravitational wave detectors, which are expected to encounter new
types of noises as they undergo gradual improvements to attain design sensitivity.

PACS numbers: 07.05.Mh, 07.05.Kf, 04.80.Nn, 95.55.Ym

Keywords: Deep Learning, Machine Learning, LIGO, Gravitational Waves, Gravity
Spy, Glitch Classification, Detector Characterization, Transfer Learning, Convolutional
Neural Network, Feature Extraction, Unsupervised Clustering

Deep Transfer Learning for aLIGO Detector Characterization

2

1. Introduction
The advanced Laser Interferometer Gravitational wave Observatory (aLIGO) detectors
are the largest and most sensitive interferometric detectors ever built. The cutting-edge
design and fabrication of aLIGO’s subsystems enable the sensing of changes in aLIGO’s
arm-length thousands of times smaller than the diameter of a proton [1, 2]. This
instrument has already detected multiple gravitational wave (GW) signals produced
from mergers of black holes [3, 4, 5]. As aLIGO gradually attains design sensitivity, it
will transition from its current discovery mode into an astronomical observatory that
will routinely detect new GW sources, providing insights into astrophysical objects and
processes that cannot be observed through any other means [6, 7].
The study of glitches is of paramount importance to characterize GW detectors. For
aLIGO to realize its full potential, it is necessary to ensure that its sensing capabilities
are not hindered by unwanted noises that contaminate GW data. This is a nontrivial task requiring “intelligent” algorithms given that noise transients vary widely
in duration, frequency range and morphology, spanning a wide parameter space that is
challenging to model accurately [8, 9, 10, 11]. Furthermore, since the aLIGO detectors
are undergoing commissioning between each observing run, we expect that new types
of glitches will be identified as aLIGO attains design sensitivity [2, 12, 13]. Accurately
characterizing glitches is essential to find and eliminate noise transients that can obscure
GW signals, or accidentally lead to coincident false detections which mimic GW signals,
due to their high occurrence rate.
The complex and time-evolving nature of glitches makes them an ideal case study
to apply machine learning algorithms — methods that learn from examples rather than
being explicitly programmed. Machine learning can be divided into supervised and
unsupervised learning depending on whether labeled, structured data is used for training
the algorithms [14]. Deep learning, i.e, machine learning based on deep artificial neural
networks [15, 14], is one of the fastest growing fields of artificial intelligence research
today, having outperformed competing methods in many areas of machine learning
applications, e.g., image classification, face recognition, natural language understanding
and translation, speech recognition and synthesis, game-playing (e.g., Go, Poker), and
self-driving vehicles. Therefore, deep learning algorithms are also expected to have
excellent performance for the characterization of gravitational wave detectors. In this
article, we focus on deep learning with Convolutional Neural Networks (CNNs), the
leading approach for computer visions tasks [15], using spectrograms computed from
the time-series glitches as inputs.
Recent efforts on this front include Gravity Spy, an innovative interdisciplinary
project that provides an infrastructure for citizen scientists to label datasets of glitches
from aLIGO via crowd-sourcing [16]. Supervised classification algorithms based on this
dataset were presented in the first Gravity Spy article [16] and were further discussed in
Ref. [17]. In the latter study, deep multi-view models were introduced to enhance
classification accuracy using multiple-duration spectrograms of each glitch stitched-

Deep Transfer Learning for aLIGO Detector Characterization

3

together to form a larger input image. These algorithms employed deep learning CNN
models which were 4 layers deep, and achieved overall accuracies close to 97% for glitch
classification. However, it was found that glitch classes with very few labeled samples
were more difficult to classify with the same level of accuracy.
As new classes of glitches are uncovered in the near future, the design and
hyperparameters ‡ of these CNNs will have to be modified accordingly to distinguish
subtle features between a larger set of classes. Considering these issues, one may be
interested in exploring the use of very deep CNNs designed for the harder task of
classifying thousands of categories of real-world objects. However, the small size of
the Gravity Spy dataset, compared to the datasets used to train these state-of-the-art
CNNs for real-world object recognition, would lead to poor results due to immediate
overfitting, i.e., memorization of features in the training data without generalizing to
the testing data. Therefore, previous attempts at glitch classification from spectrograms
have used relatively shallow CNNs, which were designed and trained from the groundup [16, 17]. With this approach, lower input image resolution is used to keep model
size manageable while significant time and effort is needed to optimize the architecture
(hyperparameters) of the CNNs and to train them from scratch. To directly employ
deep CNN models with well-established architectures optimized for computer vision, for
this glitch classification problem, a different approach is necessary.
In this article, we present Deep Transfer Learning, a new method for glitch
classification that leverages the complex structure and deep abstraction of pre-trained
state-of-the-art CNNs used for object recognition and fine-tunes them throughout all
layers to accurately classify glitches after re-training on a small dataset of aLIGO
spectrograms. We show that this technique achieves state-of-the-art results for glitch
classification with the Gravity Spy dataset, attaining above 98.8% overall accuracy and
perfect precision-recall on 8 out of 22 classes, while significantly reducing the training
time to a few minutes. We also demonstrate that features learned from real-world images
by very deep CNNs are directly transferable for the classification of spectrograms of timeseries data from GW detectors, and possibly also for spectrograms in general, although
the two datasets are very dissimilar. The CNNs we use were originally designed for over
1000 classes of objects in ImageNet. Therefore, our algorithm can be easily extended to
classify hundreds of new classes of glitches in the future, especially since this transfer
learning approach requires only a few labeled examples of a new class. Furthermore, we
outline how new classes of glitches can be automatically found and grouped together
as they occur using our trained CNNs as feature extractors for unsupervised or semisupervised clustering algorithms.
This article is organized as follows. Section 2 describes the methods and datasets
used for applying transfer learning to develop accurate CNNs for glitch classification.
Section 3 summarizes our results. We discuss immediate applications of our method and
‡ Hyperparameters refer to several quantities that have to be manually chosen to determine the
architecture of the neural networks (e.g., overall design, number of layers, sizes of the convolutions,
depth, padding, types of layers and activations, etc.)

Deep Transfer Learning for aLIGO Detector Characterization

4

its extension for finding new classes of glitches in an unsupervised manner in Section 4.
Conclusions and a brief description of future strategies to assist ongoing efforts for
aLIGO data analysis are provided in Section 5.
2. Methods
In this Section we provide a succinct description of neural networks and transfer learning.
Thereafter, we present our methods and procedure for applying transfer learning in the
context of glitch classification.
2.1. Neural Networks
CNNs are currently the best performing method for image classification, object
recognition, and a variety of other image processing tasks [15]. They serve as excellent
feature extractors by allowing end-to-end learning of features and representations from
raw image data for classification, thus eliminating the need for feature engineering,
i.e., hand-extraction of features or representations by human experts. CNN-based
algorithms have recently achieved super-human results in the ImageNet Large Scale
Visual Recognition Competition (ILSVRC) [18], and has been outperforming all other
methods since 2012 [19].
State-of-the-art CNNs for real-world object recognition are extremely powerful,
and thus training them from the ground up on a small dataset would lead to poor
performance. This is because the capacity of the model, i.e., the degrees of freedom
(number of parameters) of the model would be much larger than necessary to fit the
small data distribution, thus resulting in what is known as “overfitting”(see section 5.2
in the Deep Learning book [14]), where the algorithm memorizes the training samples
without generalizing to the test set. The symptoms of overfitting can be alleviated by
using regularization techniques, such as dropout and by limiting the number of training
iterations (early stopping). However, in practice, an extremely large labeled dataset
(typically a million samples) is required to train these randomly initialized deep CNNs
for optimal performance.
The Gravity Spy crowd-sourcing project mobilizes citizen scientists to hand-label
spectrograms obtained from aLIGO time-series data. The rationale for this approach is
that humans are capable of distinguishing different classes of noise from spectrograms
after being shown only a few examples, which indicates that generic pattern recognizers
developed in humans for real-world object recognition are also useful when distinguishing
spectrograms of glitches. Therefore, this motivated us to apply a similar approach
where a CNN is first trained on a large database of labeled real-world objects, and
this knowledge is subsequently transferred by re-training (fine-tuning) the CNN on the
dataset of spectrograms. In machine learning literature, this idea is referred to as
transfer learning, which we describe in the following Section.

Deep Transfer Learning for aLIGO Detector Characterization

5

2.2. Transfer Learning
Transfer learning is an essential ingredient for artificial intelligence, where knowledge
learned in one domain for some task can be transferred to another domain for a
different task [20]. In the context of deep learning, transfer learning is commonly
implemented by pre-training a deep neural network, e.g., a CNN, on a large labeled
dataset followed by fine-tuning (continued training) on a different dataset of interest,
which is usually smaller. This approach has been successfully applied in many areas of
computer vision [21].
It is well known that the initial layers of a CNN always learn to extract simple
generic features (e.g., edges, corners, curves, color blobs, etc.), which are applicable to
all types of images, whereas the final layers represent highly abstract and data-specific
features [21]. Therefore, using a model optimally-trained on a large dataset and then
fine-tuning it on a different dataset is expected to result in higher accuracy and a faster
training process, compared to training the same CNNs from scratch, due to the shared
features present in the initial layers.
To demonstrate the power of transfer learning for classifying glitches, we compare
the performance of the most popular CNN models for object recognition, namely
Inception [22] version 2 and 3, ResNet [23], and VGG [24], all of which were leading
entries in recent ILSVRC competitions. These CNNs were optimally trained on a large
dataset of images — i.e., ImageNet [25], which contains 1.2 million labeled images of
real-world objects belonging to 1000 categories (see Figure B4 for sample images) —
over the course of 2 to 3 weeks using multiple GPUs by other research groups. We
obtained the open-source weights from these models, and used them to initialize the
CNNs, before fine-tuning each model on the dataset of glitches.
We show that transfer learning allows us to apply these powerful CNNs for
classifying glitches using a very small training dataset of spectrograms to obtain
significantly higher accuracies, reduce the training time by several orders of magnitude,
and eliminate the need for optimizing hyper-parameters. Furthermore, we show how
information from multiple duration spectrograms can be efficiently encoded into a single
image in different color channels to enhance information provided to these CNN models.
2.3. Dataset
The Gravity Spy dataset, from the first observing run of LIGO, contains labeled
spectrogram samples from 22 classes of glitches shown in Figure 1. Within the dataset,
each sample contains 4 images recorded with durations: 0.5s, 1.0s, 2.0s, and 4.0s. These
images were hand-labeled by citizen scientists participating in the Gravity Spy project,
and the accuracy of the labeling was greatly enhanced by cross-validation techniques
within the Gravity Spy infrastructure.
We cropped each image closely around its frame, removing axis labels and whitespaces. Subsequently, the resolution of these images were 481 × 575 × 3, with the
third dimension corresponding to the RGB color channels. For training and testing,

Deep Transfer Learning for aLIGO Detector Characterization

6

Figure 1: Classes of glitches in the Gravity Spy dataset from the first observing run
of aLIGO. The No Glitch class is omitted in the figure. For each glitch sample,
spectrograms with durations of 0.5s, 1.0s, 2.0s, and 4.0s are available. The objective is
to predict the classes given the images.

these images were down-sampled to the optimal resolution for each CNN model, either
224×224×3 or 299×299×3. We used the recent version of the Gravity Spy dataset as of
April 19, 2017, which has been slightly modified since the previous publications [16, 17],
with two additional classes, namely 1080 Lines and 1400 Ripples, and about 300 extra
elements in the Violin Mode class. We randomly split this dataset, containing about
8500 elements, into two parts such that approximately 80% of samples in each class was
in the training set, and 20% of each class was in the testing set.
We did not use a separate validation set, since we did not engage in any kind of
hyperparameter optimization that could have led to overfitting on the test set. This
allowed us to use a larger fraction of the data for training and testing. We directly
used the same hyperparameters as the pre-trained models. For cross-verification of
the performance, the Inception models were trained independently from the rest of the
models, using a different deep learning software framework and different randomized
test-train splits of the data.
For training VGG16, VGG19, and ResNet50, we only used spectrogram images
with 2.0s duration, since they contained sufficient information to resolve most of the
long and short duration glitches. Since the dataset is small, image augmentation was
used to artificially enhance the size of the dataset (except when training the Inception
models). For each input image, we subtracted the mean of the training samples, then
randomly shifted it by 0 to 10 pixels in horizontal and vertical directions, and randomly
zoomed in and out with factors up to 0.05×. The mean image of all the training samples
was also subtracted from each sample used for testing.
For the Inception models, we found that we could improve the accuracy, without
any image augmentation, by encoding multiple-duration spectrograms of each glitch into

Deep Transfer Learning for aLIGO Detector Characterization

7

each of the RGB color channels to present maximum information to the CNNs. The 1.0s,
2.0s, and 4.0s duration spectrograms were converted to gray-scale, and then merged to
produce RGB images with each channel encoding the different durations. Examples
of these encoded images are shown in Figure B3. While previous work had used
concatenated or parallel-views of multiple-duration spectrograms as inputs to specially
designed CNNs trained from scratch [17], our encoding method has the advantage of
allowing the direct use of pre-trained CNN architectures designed for ImageNet for
classification of multiple-duration spectrograms.
2.4. Training
The final fully-connected layer in each CNN model was replaced with another fullyconnected layer having 22 neurons corresponding to each glitch class. To reduce
overfitting, we added several dropout layers [26] (which randomly turn neurons off to
prevent co-adaptation between neurons and thus improves their ability to generalize) at
the beginning of the network and right before the output layers. Apart from this, we
also added one additional layer after the first trainable convolutional layer for ResNet.
The details about network structures are listed in Table B1 of Appendix A. The softmax
function is used as the final layer in each model to provide probabilities of each class as
the outputs.
When training each model, we utilized the standard cross entropy loss (cost)
function due to its good performance in classification problems in combination with
the softmax layer [14]. We fine-tuned across all the layers since the dataset of glitches
is very different from the objects in the ImageNet data. Each model was trained up to
100 epochs with the usual iterative procedure [14] and check-pointed after each epoch.
The best performing check-points were chosen subsequently.
We used the standard AdaGrad [27] and ADAM [28] optimizers as the learning
algorithm [14] for training. Specifically, we used AdaGrad for ResNet, VGG16 and
VGG19 with initial learning rate set to 10−4 and  = 10−6 and the ADAM method
with parameters β1 = 0.9 and β2 = 0.999 for the Inception models. Inception V2
and V3 models were implemented in the Wolfram Language [29], which internally uses
the open-source MXNet [30] framework for training. VGG and ResNet models were
implemented and trained independently on a different randomized train-test split using
TensorFlow [31] via Keras [32].
3. Results
A naive implementation of a logistic regression classifier on top of features extracted from
the spectrograms using Inception models pre-trained on ImageNet, produced over 95%
accuracy without fine-tuning any layers. This strongly indicated that features learned
for real-world object recognition are also directly useful for classification of spectrograms
of LIGO noise transients, and thus offered incentive to now train (fine-tune) every layer.

Deep Transfer Learning for aLIGO Detector Characterization

8

The results after fine-tuning are presented below.
For the CNNs implemented in this paper, both InceptionV2 and InceptionV3
achieved over 98% accuracy in fewer than 10 epochs of training (less than 20 minutes),
VGG16 and VGG19 achieved over 98% accuracy within 30 epochs of training. However,
ResNet50 took much longer to converge. We compare the results of these CNNs trained
with the transfer learning method with the CNN models trained from scratch [16, 17] in
Table 1. We found that each of our models consistently achieved over 98% accuracy for
many epochs, thus indicating that the performance is robust, regardless of the stopping
criteria, and therefore the model is not overfitting on the test set.
The precision and recall obtained with each model on every class is reported
in Appendix A. With InceptionV3, we achieved perfect precision and recall on 8
classes: 1080Lines, 1400Ripples, Air Compressor, Chirp, Helix, Paired Doves,
Power Line, and Scratchy. With ResNet50, we achieved perfect precision and recall on
7 classes: 1080Lines, 1400Ripples, Extremely Loud, Helix, Paired Doves, Scratchy,
and Violin Mode.
Both ResNet50 and InceptionV3 achieved the highest accuracy of 98.84% on the
test set despite being trained independently via different methods on different splits
of the data with 2.0s scans and RGB encoded scans respectively. They also obtained
100% accuracy when considering the top-5 predictions, which implies that given any
input, the true class can be narrowed down to within 5 classes with 100% confidence.
This is particularly useful, since the true class of a glitch is often ambiguous to even
human experts. We have shown the confusion matrices for ResNet50 and InceptionV3
in Figure 2 and Figure 3, respectively. Confusion matrices of the remaining models
are included in Appendix A. The accuracies of each of these models, and previously
published CNN models trained from scratch on the same dataset are reported in Table 1.
Note that each of our models consistently under-performed when trained with random
initializations, i.e., without applying transfer learning. Therefore, it is evident that the
networks trained with the transfer learning approach achieve better results than the
CNN models trained from the ground up on the same dataset.
4. Discussion
The results presented in Table 1 indicate that transfer learning with state-of-the-art
image classification CNNs achieves optimal performance, while significantly reducing
the training time to less than an hour, and eliminating the difficulties associated with
hyperparameter tuning. Employing CNN models with very deep and complex structures
is feasible with this approach on very small datasets, since the larger training set used
for pre-training enables learning good feature extractors in the initial layers, and thus
prevents overfitting before learning good features. In addition, the CNNs were able to
learn high level features specific to the small dataset of glitches during the fine-tuning
process throughout all layers, which further improved the classification results.
We have also provided a comparison between the most popular state-of-the-art

Deep Transfer Learning for aLIGO Detector Characterization

9

Figure 2: Confusion matrix of ResNet50 after fine-tuning on the dataset of glitches.
The accuracy is 98.84%. Note that ResNet is the current state-of-the-art CNN model
for a variety of image processing tasks including object recognition.

models for image classification on the transfer learning problem. Furthermore, we have
demonstrated that transfer learning always provides significant improvements over the
traditional approach used for classifying spectrograms, and allows the use of the most
powerful CNNs on very small datasets, even when they are very dissimilar compared
to the original data used for pre-training. Once trained, inference (evaluation on
new inputs) can be carried out in a few milliseconds using a GPU, allowing real-time
classification of glitches in aLIGO.
Note that both the ResNet and Inception models achieved perfect precision and

Deep Transfer Learning for aLIGO Detector Characterization

10

Figure 3: Confusion matrix for InceptionV3. The accuracy is also 98.84%. It can be
seen that different errors were made compared to ResNet50. Note that the Chirp glitch
class (which is also the shape for true GW signals from mergers of binaries) and the
Paired Doves class (which only had 24 training examples) was identified with perfect
precision and recall.

Deep Transfer Learning for aLIGO Detector Characterization

11

Table 1: Accuracy on the Test Set

Neural Network

Top-1(20 classes)

Top-1

Top-2

Top-3

Top-4

Top-5

Tuned-VGG16
Tuned-VGG19
Tuned-ResNet50
Tuned-InceptionV2
Tuned-InceptionV3
CNN in [16, 17]

98.02%
98.08%
98.76%
98.70%
98.76%
96.72%

98.15%
98.21%
98.84%
98.78%
98.84%
96.70%

99.36%
99.31%
99.71%
99.59%
99.71%
98.32%

99.71%
99.60%
99.83%
99.71%
99.88%
99.13%

99.83%
99.71%
99.94%
99.94%
99.94%
99.31%

99.88%
99.71%
100%
100%
100%
99.36%

The table lists the top-1 to top-5 accuracies for different CNNs on the testing set. The top-1 accuracy
for 20 classes (excluding 1080Lines and 1400Ripples) is also shown for comparison with previous
publications. We re-trained the 4 layer merged-view CNN model described in the publications [16, 17]
from scratch on our same train-test dataset for a fair comparison, since the Gravity Spy dataset has
been recently updated. Note that our Inception and ResNet models are capable of narrowing down any
input to within 5 classes with 100% accuracy.

recall on the Paired Doves test set even though it was the smallest class with only
21 elements in the training set. Previous methods trained from the ground up had
achieved sub-optimal results for this class [16]. This analysis strongly suggests that the
high accuracy is due to transfer learning from the larger dataset, since similar patterns
could have been learned when pre-trained on the ImageNet data. Therefore, we have
shown that, although the number of samples vary greatly between different classes (up
to roughly a factor of 100), we could obtain high precision and recall on even the smallest
classes without employing any class balancing techniques for training.
The best performing models on ImageNet are currently based on ensembles/committees of different CNNs. Since the confusion matrices indicate that each
CNN has different strengths and weaknesses for many classes, we expect that using
an ensemble of different models would further boost the accuracy for glitch classification. Furthermore, as the Gravity Spy dataset is enlarged in the future with more
hand-labeled glitches, our CNNs may be re-trained with the largest available dataset to
improve the accuracy.
However, there may be an upper bound set by the error-rate of humans providing
the labels. It can be seen in Figure A4 that most of the misclassification made by our
CNNs were either due to incorrect labeling, or due to superposition of two different
classes of glitches, or due to inputs whose true class remain ambiguous. Since our
method can be trained very quickly, it can be used to efficiently correct mislabeled
elements in the original dataset by training on a large number of randomized train-test
splits, and searching for the most commonly mislabeled elements among all these test
sets.
The CNNs we have developed in this study may be immediately used in LIGO
pipelines for classifying known categories of glitches with high accuracy in real-time. The
well-known ability of deep neural networks to generalize implies that the classifications

Deep Transfer Learning for aLIGO Detector Characterization

12

would be resilient to changes in background noise. Therefore, we expect the excellent
performance we achieved on the current data would translate to future observing runs.
Nonetheless, the accuracy can be further improved by periodically re-training with larger
datasets containing all available labeled examples of glitches at each time.
After applying transfer learning, i.e., fine-tuning the CNNs on the small dataset
of glitches, the CNNs may also be used as good feature extractors for finding new
categories of glitches from unlabeled data in an unsupervised or semi-supervised manner.
This method, which combines supervised learning, transfer learning, and unsupervised
learning, can be used to identify many more categories of noise transients and estimate
at what times new types of glitches with similar morphologies start occurring. This may
also be used to correct mislabeled glitches in the original dataset used for training/testing
by searching for anomalies in the feature-space.
For instance, consider our InceptionV3 model: removing the final softmax and fullyconnected layer near the output produces a CNN that maps any input image to a 1008dimensional vector. In this high-dimensional space, glitches having similar morphology
will be clustered together. Anomalies and mis-labeled examples will appear isolated
from the clusters. This can be visualized by reducing the dimensions of the feature
space to 2-D or 3-D using a suitable dimension reduction algorithm (e.g., t-SNE [33])
as shown in Figure C1 and Figure C2).
Therefore, when new types of glitches appear, which are classified as
None of the Above by our CNN model, they may be mapped to vectors using these
truncated CNN feature extractors. Then, new classes of glitches may be identified by
applying standard unsupervised clustering algorithms (see Appendix C for a proof-ofconcept). This automated clustering can also be used to accelerate human labeling of
new glitches in a semi-supervised manner by presenting batches of similar glitches to a
citizen scientist.
We believe the reason why our clustering technique works very well is because
we have used very deep CNNs that are capable of extracting and representing a large
number of highly abstract features of the input images. Therefore, transfer learning
enables the use of these CNNs which are excellent feature extractors, thus effectively
enabling the use of unsupervised learning techniques for finding new classes of glitches.
The performance of different clustering methods may be investigated in detail in a
subsequent study.
5. Conclusion
This article is part of a program that aims to spread the use of deep learning in
many related areas of multimessenger astrophysics [34], ranging from detection and
characterization of GW sources, to detector characterization. This is the first application
of Deep Transfer Learning in the context of aLIGO. Using this method, we have
developed a state-of-the-art DNNs for aLIGO glitch classification using the Gravity
Spy dataset. We have shown that by combining the pre-trained weights of state-of-

Deep Transfer Learning for aLIGO Detector Characterization

13

the-art CNNs with fine-tuning across all layers, we can further increase the accuracy
compared to neural networks that have been trained from scratch using only the Gravity
Spy dataset. Furthermore, we have shown that the training time is significantly reduced
with our approach when compared to traditional methods and the effort required to
design CNN models and optimize hyper-parameters can be eliminated.
Note that even though the Gravity Spy dataset of spectrograms was not at all
similar to the real-world objects found in the ImageNet database, we have demonstrated
that the transfer learning approach works extremely well. Therefore, we expect that our
method will be useful for time-series classification in general, where the problem can
be represented in the form of images. Furthermore, we have developed a novel method
to encode different duration spectrograms (i.e., multiple views of the same input) as
the different RGB color channels in a single image, which renders excellent results and
allows the direct use of pre-trained state-of-the-art CNN models.
The algorithms we have introduced in this article may be used to classify new
time-series data in the Gravity Spy project, as well as data from future aLIGO
observing runs and data from international partners, such as Virgo [35], KAGRA [36]
and LIGO-India [37] as they come online in the next few years. This would be useful
both for improving the data quality the detectors and for generating vetoes in GW
search pipelines. The transfer learning method allows us to use very deep CNNs, with
well-known architectures, which are capable of extracting more complex and abstract
features from the spectrogram images. New classes of glitches may also be found in an
unsupervised manner or labeled rapidly in a semi-supervised manner with our method by
truncating these CNNs, and using them as feature extractors for clustering algorithms.
By employing this transfer learning method for glitch classification, we can create a
larger dataset of labeled transients with real aLIGO data, and the corresponding timeseries can be fetched from the GPS times to produce labeled time-series inputs. This
dataset may be added to the training process in the Deep Filtering pipeline introduced
in [34], to unify detection, classification, and parameter estimation directly from timeseries data streams for GW detectors in real-time. The fully-trained CNNs developed in
this article will be made available to the community for use in glitch classification and
detector characterization pipelines.
6. Acknowledgements
We are grateful to NVIDIA for supporting this research with GPU hardware donations,
to Wolfram Research for offering several Mathematica licenses, and to Vlad Kindratenko
for providing dedicated access to a machine at the Innovative Systems Lab at NCSA.
We acknowledge the Gravity Spy project and the citizen scientists who participated
in it for creating the dataset we used. We thank Gabrielle Allen, Scott Coughlin,
Vicky Kalogera, Joshua Smith, Kai Staats, Sara Bahaadini, and Michael Zevin for
productive interactions. We thank Kai Staats, Laura Nuttall, and the Gravity Spy
team for reviewing this article and providing feedback. We also acknowledge the LIGO

Deep Transfer Learning for aLIGO Detector Characterization

14

collaboration for the use of computational resources and for conducting the experiments
from which the raw data was obtained. This document has LIGO number P1700128.
[1] The LIGO Scientific Collaboration, Aasi J et al. 2015 Classical and Quantum Gravity 32 074001
(Preprint 1411.4547)
[2] The LIGO Scientific Collaboration and The Virgo Collaboration 2016 ArXiv e-prints (Preprint
1602.03838)
[3] Abbott B P e a (LIGO Scientific Collaboration and Virgo Collaboration) 2016 Phys. Rev. Lett.
116(6) 061102 URL http://link.aps.org/doi/10.1103/PhysRevLett.116.061102
[4] Abbott B P e a (LIGO Scientific Collaboration and Virgo Collaboration) 2016 Phys. Rev. Lett.
116(24) 241103 URL http://link.aps.org/doi/10.1103/PhysRevLett.116.241103
[5] Abbott B P, Abbott R, Abbott T D, Abernathy M R, Acernese F, Ackley K, Adams C, Adams
T, Addesso P, Adhikari R X and et al 2017 Physical Review Letters 118(22) 221101 URL
https://link.aps.org/doi/10.1103/PhysRevLett.118.221101
[6] LIGO Scientific Collaboration, Virgo Collaboration, Aasi J, Abadie J, Abbott B P, Abbott
R, Abbott T D, Abernathy M, Accadia T, Acernese F et al. 2013 ArXiv e-prints (Preprint
1304.0670)
[7] Sathyaprakash B S and Schutz B F 2009 Living Reviews in Relativity 12 2 (Preprint 0903.0338)
[8] Cornish N J and Littenberg T B 2015 Classical and Quantum Gravity 32 135012 (Preprint
1410.3835)
[9] Powell J, Trifirò D, Cuoco E, Heng I S and Cavaglià M 2015 Classical and Quantum Gravity 32
215012 (Preprint 1505.01299)
[10] Powell J, Torres-Forné A, Lynch R, Trifirò D, Cuoco E, Cavaglià M, Heng I S and Font J A 2016
ArXiv e-prints (Preprint 1609.06262)
[11] Mukund N, Abraham S, Kandhasamy S, Mitra S and Philip N S 2017 Phys. Rev. D 95(10) 104059
URL https://link.aps.org/doi/10.1103/PhysRevD.95.104059
[12] The LIGO Scientific Collaboration and the Virgo Collaboration 2016 ArXiv e-prints (Preprint
1602.03844)
[13] The LIGO Scientific Collaboration and Abbott B P 2016 ArXiv e-prints (Preprint 1602.03845)
[14] Goodfellow I, Bengio Y and Courville A 2016 Deep Learning (MIT Press)
[15] Lecun Y, Bengio Y and Hinton G 2015 Nature 521 436–444 ISSN 0028-0836
[16] Zevin M, Coughlin S, Bahaadini S, Besler E, Rohani N, Allen S, Cabero M, Crowston K,
Katsaggelos A, Larson S, Lee T K, Lintott C, Littenberg T, Lundgren A, Oesterlund C, Smith
J, Trouille L and Kalogera V 2017 Classical and Quantum Gravity 34 (Preprint 1611.04596)
[17] Bahaadini S, Rohani N, Coughlin S, Zevin M, Kalogera V and Katsaggelos A K 2017 International
Conference on Acoustics, Speech and Signal Processing (Preprint 1705.00034)
[18] Deng J, Dong W, Socher R, Li L J, Li K and Fei-Fei L 2009 Imagenet: A large-scale hierarchical
image database Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference
on (IEEE) pp 248–255
[19] Krizhevsky A, Sutskever I and Hinton G E 2012 Imagenet classification with deep convolutional
neural networks Advances in Neural Information Processing Systems 25 ed Pereira F, Burges
C J C, Bottou L and Weinberger K Q (Curran Associates, Inc.) pp 1097–1105
[20] Yosinski J, Clune J, Bengio Y and Lipson H 2014 How transferable are features in deep neural
networks? Proceedings of the 27th International Conference on Neural Information Processing
Systems NIPS’14 (Cambridge, MA, USA: MIT Press) pp 3320–3328 URL http://dl.acm.org/
citation.cfm?id=2969033.2969197
[21] Sharif Razavian A, Azizpour H, Sullivan J and Carlsson S 2014 ArXiv e-prints (Preprint
1403.6382)
[22] Christian S, Wei L, Yangqing J, Pierre S, Scott R, Dragomir A, Dumitru E, Vincent V and Andrew
R 2014 ArXiv e-prints (Preprint 1409.4842)

Deep Transfer Learning for aLIGO Detector Characterization
[23]
[24]
[25]
[26]
[27]
[28]
[29]
[30]
[31]

[32]
[33]
[34]
[35]
[36]
[37]
[38]

15

Kaiming H, Xiangyu Z, Shaoqing R and Jian S 2015 ArXiv e-prints (Preprint 1512.03385)
Karen S and Andrew Z 2014 ArXiv e-prints (Preprint 1409.1556)
Deng J, Dong W, Socher R, Li L J, Li K and Fei-Fei L 2009 CVRP
Srivastava N, Hinton G, Krizhevsky A, Sutskever I and Salakhutdinov R 2014 Journal of Machine
Learning Research 15 1929–1958 URL http://jmlr.org/papers/v15/srivastava14a.html
John D, Elad H and Yoram S 2011 Journal of Machine Learning Research 12
Kingma D P and Ba J 2014 CoRR abs/1412.6980 URL http://arxiv.org/abs/1412.6980
Wolfram Language System & Documentation Center https://reference.wolfram.com/
language/
Chen T, Li M, Li Y, Lin M, Wang N, Wang M, Xiao T, Xu B, Zhang C and Zhang Z 2015 CoRR
abs/1512.01274 URL http://arxiv.org/abs/1512.01274
Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G,
Isard M, Kudlur M, Levenberg J, Monga R, Moore S, Murray D G, Steiner B, Tucker P,
Vasudevan V, Warden P, Wicke M, Yu Y and Zheng X 2016 Tensorflow: A system for largescale machine learning Proceedings of the 12th USENIX Conference on Operating Systems Design
and Implementation OSDI’16 (USENIX Association) pp 265–283 ISBN 978-1-931971-33-1 URL
http://dl.acm.org/citation.cfm?id=3026877.3026899
Chollet F et al. 2015
Maaten L v d and Hinton G 2008 Journal of Machine Learning Research 9 2579–2605
George D and Huerta E A 2017 ArXiv e-prints (Preprint 1701.00008)
Acernese F et al. 2015 Classical and Quantum Gravity 32 024001 (Preprint 1408.3978)
Hirose E, Sekiguchi T, Kumar R, Takahashi R and the KAGRA Collaboration 2014 Classical and
Quantum Gravity 31 224004
Unnikrishnan C S 2013 International Journal of Modern Physics D 22 1341010
The Wolfram Language Image Identification Project https://www.imageidentify.com/

Deep Transfer Learning for aLIGO Detector Characterization

16

Appendix A.
In this Section we present supplementary information summarizing the networks we
experimented with to classify spectrograms of the Gravity Spy dataset. Figure A1–
Figure A3 present the confusion matrices for VGG16, VGG19 and InceptionV2 models
respectively.

Figure A1: Confusion matrix for VGG16. The accuracy is 98.15%.
Table A1 presents the recall and precision of our Deep Transfer Learning algorithm for
glitch classification. These results clearly exhibit the power of this method, especially
for rare classes with few labeled examples, therefore the transfer learning method offers
a promising framework for future development in glitch classification for GW detectors.

Deep Transfer Learning for aLIGO Detector Characterization

Figure A2: Confusion matrix for VGG19. The accuracy is 98.21%.

17

Deep Transfer Learning for aLIGO Detector Characterization

18

Table A1: Table of Precision and Recall on the Test Set

Class
1080Lines
1400Ripples
Air Compressor
Blip
Chirp
Extremely Loud
Helix
Koi Fish
Light Modulation
Low Frequency Burst
Low Frequency Lines
No Glitch
None of the Above
Paired Doves
Power Line
Repeating Blips
Scattered Light
Scratchy
Tomte
Violin Mode
Wandering Line
Whistle

Quantity

VGG16

VGG19

ResNet50

InceptionV2

InceptionV3

CNN in [16, 17]

recall
precision
recall
precision
recall
precision
recall
precision
recall
precision
recall
precision
recall
precision
recall
precision
recall
precision
recall
precision
recall
precision
recall
precision
recall
precision
recall
precision
recall
precision
recall
precision
recall
precision
recall
precision
recall
precision
recall
precision
recall
precision
recall
precision

100%
98.51%
100%
97.92%
100%
80.00%
99.47%
97.66%
100%
100%
100%
98.91%
100%
100%
99.40%
99.41%
99.13%
97.44%
96.97%
98.46%
100%
97.49%
83.78%
100%
61.11%
78.57%
100%
100%
97.80%
100%
94.74%
100%
97.83%
100%
100%
100%
95.83%
95.83%
100%
100%
88.89%
88.89%
96.72%
98.33%

100%
98.50%
100%
97.91%
100%
85.71%
98.67%
98.94%
100%
100%
100%
98.91%
100%
100%
99.40%
99.40%
100%
95.04%
96.97%
99.22%
100%
95.79%
86.49%
100%
50.00%
69.23%
83.33%
100%
98.90%
100%
96.49%
94.83%
100%
100%
100%
100%
95.83%
92.00%
100%
100%
100%
90.00%
96.72%
100%

100%
100%
100%
100%
100%
85.71%
98.67%
99.20%
100%
93.33%
100%
100%
100%
100%
98.80%
100%
100%
99.14%
98.48%
98.48%
100%
95.79%
91.89%
100%
83.33%
93.75%
100%
100%
97.80%
100%
98.25%
100%
98.91%
100%
100%
100%
100%
82.76%
100%
100%
100%
90.00%
98.36%
100%

100%
95.65%
100%
100%
100%
100%
99.99%
98.94%
92.86%
100%
100%
98.91%
100%
100%
98.80%
98.80%
97.39%
99.11%
100%
95.65%
97.80%
100%
100%
100%
66.67%
92.31%
100%
100%
98.90%
100%
96.49%
100%
98.91%
98.91%
100%
100%
100%
92.31%
100%
100%
77.78%
100%
98.36%
100%

100%
100%
100%
100%
100%
100%
99.73%
99.73%
100%
100%
100%
97.85%
100%
100%
99.40%
99.40%
95.65%
98.21%
99.24%
97.04%
100%
97.85%
97.30%
100%
61.11%
91.67%
100%
100%
100%
100%
98.25%
96.55%
98.91%
100%
100%
100%
100%
88.89%
100%
97.94%
88.89%
100%
98.36%
100%

100%
97.06%
91.49%
82.69%
100%
85.71%
98.94%
98.15%
100%
100%
100%
100%
98.21%
94.83%
100%
99.40%
99.13%
98.28%
96.21%
100%
95.60%
92.55%
86.49%
94.12%
44.44%
80.00%
100%
100%
98.90%
100%
89.47%
94.44%
98.91%
97.85%
100%
97.26%
100%
85.71%
97.89%
95.88%
66.67%
75.00%
83.61%
94.44%

The table lists recall and precision for different CNN models. The last column compares
the results with the merged-view CNN model described in [16, 17] after training and
testing from scratch on the same train-test split we used for our models with the updated
Gravity Spy dataset.

Deep Transfer Learning for aLIGO Detector Characterization

Figure A3: Confusion matrix for InceptionV2. The accuracy is 98.78%.

19

Deep Transfer Learning for aLIGO Detector Characterization

20

Figure A4: All misclassified elements in the test set for the InceptionV3 model. The 1s
and 4s spectrograms are shown. It can be seen that the CNNs always outputs reasonable
classifications with many of the mistakes due to incorrect labels in the test set or glitches
whose classes appear ambiguous to even humans. Therefore, the CNN has achieved close
to ideal accuracy, i.e, the Bayes error rate. This suggests that mislabeled examples can
be found and corrected by trying different randomized test-train splits of the dataset,
and then inspecting the mistakes made by the CNN on the test set.

Deep Transfer Learning for aLIGO Detector Characterization

21

Appendix B.
Table B1 describes in detail the architecture of VGG16, VGG19 and ResNet50, including
a snapshot of the final structure of these networks. The more complex architecture of
the Inception model is shown in Figure B1.
Table B1: Neural Network Structures
VGG16

VGG19

ResNet50

1 × 1, 64
 3 × 3, 64  × 3
1 × 1, 256


1 × 1, 128
 3 × 3, 128  × 3
1 × 1, 512


1 × 1, 256
 3 × 3, 256  × 6
1 × 1, 1024


1 × 1, 512
 3 × 3, 512  × 3
1 × 1, 2048
Average-Pooling


Conv3-64 × 2

Conv3-64 × 2

Conv3-128 × 2

Conv3-128 × 2

Conv3-256 × 3

Conv3-256 × 4

Conv3-512 × 6

Conv3-512 × 8

Max-Pooling

Max-Pooling

FC-4096 × 2

FC-4096 × 2

FC-22 Softmax

FC-22 Softmax

FC-22 Softmax

VGG16, VGG19 and ResNet50 architectures are shown. The ResNet50 model employs
residual (skip) connections between the modules. The final layers of each of the networks
were replaced with a new fully connected (FC) layer with 22 neurons followed by the
softmax function.

Deep Transfer Learning for aLIGO Detector Characterization

22

Figure B1: Architecture of InceptionV2 and InceptionV3 models. The parallel structure
with Inception modules significantly improves evaluation speed and reduces total model
size compared to the other CNN models. We used the InceptionV2 model currently in
use for the ImageIdentify project pre-trained on a superset of the ImageNet dataset,
which is publicly available at [38].

Deep Transfer Learning for aLIGO Detector Characterization

conv1_1
relu1_1
conv1_2
relu1_2
pool1
conv2_1
relu2_1
conv2_2
relu2_2
pool2
conv3_1
relu3_1
conv3_2
relu3_2
conv3_3
relu3_3
pool3
conv4_1
relu4_1
conv4_2
relu4_2
conv4_3
relu4_3
pool4
conv5_1
relu5_1
conv5_2
relu5_2
conv5_3
relu5_3
pool5
flatten_0
fc6
relu6
drop6
fc7
relu7
drop7
fc8
prob

input
convolution
ReLU
convolution
ReLU
pooling
convolution
ReLU
convolution
ReLU
pooling
convolution
ReLU
convolution
ReLU
convolution
ReLU
pooling
convolution
ReLU
convolution
ReLU
convolution
ReLU
pooling
convolution
ReLU
convolution
ReLU
convolution
ReLU
pooling
flatten
linear
ReLU
dropout
linear
ReLU
dropout
linear
softmax
output

23

image
3-tensor (size: 3 × 224 × 224)
3-tensor (size: 64 × 224 × 224)
3-tensor (size: 64 × 224 × 224)
3-tensor (size: 64 × 224 × 224)
3-tensor (size: 64 × 224 × 224)
3-tensor (size: 64 × 112 × 112)
3-tensor (size: 128 × 112 × 112)
3-tensor (size: 128 × 112 × 112)
3-tensor (size: 128 × 112 × 112)
3-tensor (size: 128 × 112 × 112)
3-tensor (size: 128 × 56 × 56)
3-tensor (size: 256 × 56 × 56)
3-tensor (size: 256 × 56 × 56)
3-tensor (size: 256 × 56 × 56)
3-tensor (size: 256 × 56 × 56)
3-tensor (size: 256 × 56 × 56)
3-tensor (size: 256 × 56 × 56)
3-tensor (size: 256 × 28 × 28)
3-tensor (size: 512 × 28 × 28)
3-tensor (size: 512 × 28 × 28)
3-tensor (size: 512 × 28 × 28)
3-tensor (size: 512 × 28 × 28)
3-tensor (size: 512 × 28 × 28)
3-tensor (size: 512 × 28 × 28)
3-tensor (size: 512 × 14 × 14)
3-tensor (size: 512 × 14 × 14)
3-tensor (size: 512 × 14 × 14)
3-tensor (size: 512 × 14 × 14)
3-tensor (size: 512 × 14 × 14)
3-tensor (size: 512 × 14 × 14)
3-tensor (size: 512 × 14 × 14)
3-tensor (size: 512 × 7 × 7)
vector (size: 25088)
vector (size: 4096)
vector (size: 4096)
vector (size: 4096)
vector (size: 4096)
vector (size: 4096)
vector (size: 4096)
vector (size: 1000)
vector (size: 1000)
class

Figure B2: Architecture of VGG16. VGG19 has a similar linear design with additional
convolution modules as shown in Table B1.

Figure B2 presents the architecture of VGG16, and outlines the specific differences with
VGG19.

Deep Transfer Learning for aLIGO Detector Characterization

24

Figure B3: Random samples of multi-duration spectrograms encoded in single images.
The RGB color channels correspond to 1.0s, 2.0s, and 3.0s durations respectively. This
encoding was used to train and test the Inception models.

Figure B3 presents samples of Gravity Spy glitches used to train and test our Inception
models. Figure B4 presents a gallery of samples from ImageNet. This database is used
in the ILSVRC competitions to benchmark computer vision algorithms, and was used
to pre-train the CNN models which we utilized for glitch classification in this article.
Appendix C.

Deep Transfer Learning for aLIGO Detector Characterization

25

Figure B4: Random samples of images from the ImageNet database, which is a common
benchmark used to test computer vision algorithms. It contains images of real-world
objects belonging to 1000 different classes. Examples of classes include hamster, taxi
cab, stingray, tricycle, volleyball, lamp ,mushroom, restaurant, water bottle,
hook, ladle, kite, speedboat, etc. This dataset was used to pre-train the models used
in this article for glitch classification.

Deep Transfer Learning for aLIGO Detector Characterization

26

Figure C1: Location of different classes of glitches from the test set after applying
the CNN feature extractor. The t-SNE algorithm was used to reduce the dimension
from a 1008 vector to a 3D vector for visualization. Note that each type of glitch
forms a cluster, and their relative positions depend on their morphology. Outliers may
be inspected closely to verify whether the labels are accurate or whether they should
belong to a new class.

Figure C2: This is the CNN feature-space projected to 3D using t-SNE. A new class
called Reverse Chirp, which are chirps reflected about the vertical axis, has been added.
It can been seen that the CNN feature-extractor maps this class (which it has never seen
before during training) to a unique cluster. Furthermore, this cluster is located near the
Chirp class and the None of the Above class, which indicates that the relative positions
of the glitches in this feature-space is meaningful and depends on their morphology.

Deep Transfer Learning for aLIGO Detector Characterization

27

Figure C3: Unsupervised clustering of glitches belonging to none of the known classes.
Note that glitches having similar morphology are located near each other. Our trained
CNNs were used as feature extractors to map the images to a 1008-dimensional vectorspace, which in turn was mapped to the 2-D space shown above using the t-SNE
dimension reduction method. This feature-extractor can be used to group new classes
of glitches by morphology and find the time of their occurrence in the future.

Deep Transfer Learning for aLIGO Detector Characterization

28

Figure C4: Some clusters of glitches in the None of the Above class automatically
found by an unsupervised spectral clustering algorithm applied to the CNN featurespace shown in the previous Figure. Each row represents a cluster. Different clustering
algorithms may be used, and their parameters can be tuned to vary the sensitivity of
clustering to enforce stricter similarity criteria.

