Resnet
1023605
Inception v4, Inception ResNet and the Impact of Residual Connections on Learning
Very deep convolutional networks have been central to the largest advances in image recognition performance in recent years. One example is the Inception architecture that has been shown to achieve very good performance at relatively low computational cost. Recently, the introduction of residual connections in conjunction with a more traditional architecture has yielded state of the art performance in the 2015 ILSVRC challenge; its performance was similar to the latest generation Inception v3 network. This raises the question of whether there are any benefit in combining the Inception architecture with residual connections. Here we give clear empirical evidence that training with residual connections accelerates the training of Inception networks significantly. There is also some evidence of residual Inception networks outperforming similarly expensive Inception networks without residual connections by a thin margin. We also present several new streamlined architectures for both residual and non residual Inception networks. These variations improve the single frame recognition performance on the ILSVRC 2012 classification task significantly. We further demonstrate how proper activation scaling stabilizes the training of very wide residual Inception networks. With an ensemble of three residual and one Inception v4, we achieve 3.08 percent top 5 error on the test set of the ImageNet classification (CLS) challenge
AAAI
Christian Szegedy Sergey Ioffe Vincent Vanhoucke Alexander Amir Alemi
7507210
Wider or Deeper: Revisiting the ResNet Model for Visual Recognition
Abstract The community has been going deeper and deeper in designing one cutting edge network after another, yet some works are there suggesting that we may have gone too far in this dimension. Some researchers unravelled a residual network into an exponentially wider one, and assorted the success of residual networks to fusing a large amount of relatively shallow models. Since some of their early claims are still not settled, we in this paper dig more on this topic, i.e. the unravelled view of residual networks. Based on that, we try to find a good compromise between the depth and width. Afterwards, we walk through a typical pipeline of developing a deep learning based algorithm. We start from a group of relatively shallow networks, which perform as well or even better than the current (much deeper) state of the art models on the ImageNet classification dataset. Then, we initialize fully convolutional networks (FCNs) using our pre trained models, and tune them for semantic image segmentation. Results show that the proposed networks, as pre trained features, can boost existing methods a lot. Even without exhausting the sophistical techniques to improve the classic FCN model, we achieve comparable results with the best performers on four widely used datasets, i.e. Cityscapes, PASCAL VOC, ADE20k and PASCAL Context. The code and pre trained models are released for public access 1
Pattern Recognit.
Zifeng Wu Chunhua Shen Anton van den Hengel
214727906
COVID ResNet: A Deep Learning Framework for Screening of COVID19 from Radiographs
In the last few months, the novel COVID19 pandemic has spread all over the world. Due to its easy transmission, developing techniques to accurately and easily identify the presence of COVID19 and distinguish it from other forms of flu and pneumonia is crucial. Recent research has shown that the chest Xrays of patients suffering from COVID19 depicts certain abnormalities in the radiography. However, those approaches are closed source and not made available to the research community for re producibility and gaining deeper insight. The goal of this work is to build open source and open access datasets and present an accurate Convolutional Neural Network framework for differentiating COVID19 cases from other pneumonia cases. Our work utilizes state of the art training techniques including progressive resizing, cyclical learning rate finding and discriminative learning rates to training fast and accurate residual neural networks. Using these techniques, we showed the state of the art results on the open access COVID 19 dataset. This work presents a 3 step technique to fine tune a pre trained ResNet 50 architecture to improve model performance and reduce training time. We call it COVIDResNet. This is achieved through progressively re sizing of input images to 128x128x3, 224x224x3, and 229x229x3 pixels and fine tuning the network at each stage. This approach along with the automatic learning rate selection enabled us to achieve the state of the art accuracy of 96.23% (on all the classes) on the COVIDx dataset with only 41 epochs. This work presented a computationally efficient and highly accurate model for multi class classification of three different infection types from along with Normal individuals. This model can help in the early screening of COVID19 cases and help reduce the burden on healthcare systems.
ArXiv
Muhammad Khalid Farooq Abdul Hafeez
215917566
ResNet Attention model for human authentication using ECG signals
School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China Faculty of Computers and Information, Menoufia University, Menoufia, Egypt Department of Information and Communications Technology, Faculty of Computer Science and Telecommunications, Cracow University of Technology, Krakow, Poland Institute of Theoretical and Applied Informatics, Polish Academy of Sciences, Gliwice, Poland Department of Electronics and Computer Engineering, Ngee Ann Polytechnic, Singapore, Singapore Department of Biomedical Engineering, School of Science and Technology, Singapore School of Social Sciences, Singapore, Singapore Department of Bioinformatics and Medical Engineering, Asia University, Taichung, Taiwan
Expert Syst. J. Knowl. Eng.
Mohamed Hammad Pawel Plawiak Kuanquan Wang U Rajendra Acharya
226300360
Fighting against COVID 19: A novel deep learning model based on YOLO v2 with ResNet 50 for medical face mask detection
Deep learning has shown tremendous potential in many real life applications in different domains. One of these potentials is object detection. Recent object detection which is based on deep learning models has achieved promising results concerning the finding of an object in images. The objective of this paper is to annotate and localize the medical face mask objects in real life images. Wearing a medical face mask in public areas, protect people from COVID 19 transmission among them. The proposed model consists of two components. The first component is designed for the feature extraction process based on the ResNet 50 deep transfer learning model. While the second component is designed for the detection of medical face masks based on YOLO v2. Two medical face masks datasets have been combined in one dataset to be investigated through this research. To improve the object detection process, mean IoU has been used to estimate the best number of anchor boxes. The achieved results concluded that the adam optimizer achieved the highest average precision percentage of 81% as a detector. Finally, a comparative result with related work has been presented at the end of the research. The proposed detector achieved higher accuracy and precision than the related work.
Sustainable Cities and Society
Mohamed Loey Gunasekaran Manogaran Mohamed Hamed N Taha Nour Eldeen M Khalifa
204967424
ML ResNet: A novel network to detect and locate myocardial infarction using 12 leads ECG
BACKGROUND AND OBJECTIVE Myocardial infarction (MI) is one of the most threatening cardiovascular diseases for human beings, which can be diagnosed by electrocardiogram (ECG) Automated detection methods based on ECG focus on extracting handcrafted features. However, limited by the performance of traditional methods and individual differences between patients, it's difficult for predesigned features to detect MI with high performance. METHODS The paper presents a novel method to detect and locate MI combining a multi lead residual neural network (ML ResNet) structure with three residual blocks and feature fusion via 12 leads ECG records. Specifically, single lead feature branch network is trained to automatically learn the representative features of different levels between different layers, which exploits local characteristics of ECG to characterize the spatial information representation. Then all the lead features are fused together as global features. To evaluate the generalization of proposed method and clinical utility, two schemes including the intra patient scheme and inter patient scheme are all employed. RESULTS Experimental results based on PTB (Physikalisch Technische Bundesanstalt) database shows that our model achieves superior results with the accuracy of 95.49% the sensitivity of 94.85% the specificity of 97.37% and the F1 score of 96.92% for MI detection under the inter patient scheme compared to the state of the art. By contrast, the accuracy is 99.92% and the F1 score is 99.94% based on 5 fold cross validation under the intra patient scheme. As for five types of MI location, the proposed method also yields an average accuracy of 99.72% and F1 of 99.67% in the intra patient scheme. CONCLUSIONS The proposed method for MI detection and location has achieved superior results compared to other detection methods. However, further promotion of the performance based on MI location for the inter patient scheme still depends significantly on the mass data and the novel model which reflects spatial location information of different leads subtly.
Comput. Methods Programs Biomed.
Chuang Han Li Shi
216450330
The classification of gliomas based on a Pyramid dilated convolution resnet model
Abstract Gliomas are characterized by high morbidity and high mortality in primary tumors. The identification of glioma type is helpful for radiologists to facilitate correct medical judgments and better prognosis for patients. In order to avoid harm to patients caused by a biopsy, radiologists attempt to classify Magnetic Resonance Images(MRI) using deep learning methods. In the present paper, we propose a deep learning convolutional neural network ResNet based on the pyramid dilated convolution for Gliomas classification. The pyramid dilated convolution is integrated into the bottom of Resnet to increase the receptive field of the original network and improve the classification accuracy. After adding the pyramid dilated convolution model, the receptive field of the original network underlying convolution was improved. A clinical dataset is used to test the pyramid dilated convolution ResNet neural network model proposed in this paper. The experimental results demonstrate that the proposed method can effectively improve glioma classification performance.
Pattern Recognit. Lett.
Zhenyu Lu Yanzhong Bai Yi Chen Chun-Qiu Su Shan-Shan Lu Tianming Zhan X-N Hong Shuihua Wang
204734051
Inception and ResNet features are (almost) equivalent
Abstract Deep convolutional neural networks (CNNs) are the dominant technology in computer vision today. Much of the recent computer vision literature can be thought of as a competition to find the best architecture for vision within the deep convolutional framework. Despite all the effort invested in developing sophisticated convolutional architectures, however, it's not clear how different from each other the best CNNs really are. This paper measures the similarity between two well known CNNs, Inception and ResNet, in terms of the properties they extract from images. We find that the properties extracted by Inception are very similar to the properties extracted by ResNet, in the sense that either feature set can be well approximated by an affine transformation of the other. In particular, we find evidence that the information extracted from images by ResNet is also extracted by Inception, and in some cases may be more robustly extracted by Inception. In the other direction, most but not all of the information extracted by Inception is also extracted by ResNet. The similarity between Inception and ResNet features is surprising. Convolutional neural networks learn complex non linear features of images, and the architectural differences between systems suggest that these non linear functions should take different forms. Nonetheless, Inception and ResNet were trained on the same data set and seem to have learned to extract similar properties from images. In essence, their training algorithms hill climb in totally different spaces, but find similar solutions. This suggests that for CNNs, the selection of the training set may be more important than the selection of the convolutional architecture. keyword: ResNet, Inception, CNN, Feature Evaluation, Feature Mapping.
Cognitive Systems Research
David G McNeely-White J Ross Beveridge Bruce A Draper
91184531
Why ResNet Works? Residuals Generalize
Residual connections significantly boost the performance of deep neural networks. However, few theoretical results address the influence of residuals on the hypothesis complexity and the generalization ability of deep neural networks. This article studies the influence of residual connections on the hypothesis complexity of the neural network in terms of the covering number of its hypothesis space. We first present an upper bound of the covering number of networks with residual connections. This bound shares a similar structure with that of neural networks without residual connections. This result suggests that moving a weight matrix or nonlinear activation from the bone to a vine would not increase the hypothesis space. Afterward, an \mathcal O(1 \sqrt {N} margin based multiclass generalization bound is obtained for ResNet, as an exemplary case of any deep neural network with residual connections. Generalization guarantees for similar state of the art neural network architectures, such as DenseNet and ResNeXt, are straightforward. According to the obtained generalization bound, we should introduce regularization terms to control the magnitude of the norms of weight matrices not to increase too much, in practice, to ensure a good generalization ability, which justifies the technique of weight decay.
IEEE Transactions on Neural Networks and Learning Systems
Fengxiang He Tongliang Liu Dacheng Tao
212675182
A Mean field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth
Training deep neural networks with stochastic gradient descent (SGD) can often achieve zero training loss on real world tasks although the optimization landscape is known to be highly non convex. To understand the success of SGD for training deep neural networks, this work presents a mean field analysis of deep residual networks, based on a line of works that interpret the continuum limit of the deep residual network as an ordinary differential equation when the network capacity tends to infinity. Specifically, we propose a new continuum limit of deep residual networks, which enjoys a good landscape in the sense that every local minimizer is global. This characterization enables us to derive the first global convergence result for multilayer neural networks in the mean field regime. Furthermore, without assuming the convexity of the loss landscape, our proof relies on a zero loss assumption at the global minimizer that can be achieved when the model shares a universal approximation property. Key to our result is the observation that a deep residual network resembles a shallow network ensemble, i.e. a two layer network. We bound the difference between the shallow network and our ResNet model via the adjoint sensitivity method, which enables us to apply existing mean field analyses of two layer networks to deep networks. Furthermore, we propose several novel training schemes based on the new continuous model, including one training procedure that switches the order of the residual blocks and results in strong empirical performance on the benchmark datasets.
ICML
Yiping Lu Chao Ma Yulong Lu Jianfeng Lu Lexing Ying