high performance computing container
3334788
Container based operating system virtualization: a scalable, high performance alternative to hypervisors
Hypervisors, popularized by Xen and VMware, are quickly becoming commodity. They are appropriate for many usage scenarios, but there are scenarios that require system virtualization with high degrees of both isolation and efficiency. Examples include HPC clusters, the Grid, hosting centers, and PlanetLab. We present an alternative to hypervisors that is better suited to such scenarios. The approach is a synthesis of prior work on resource containers and security containers applied to general purpose, time shared operating systems. Examples of such container based systems include Solaris 10, Virtuozzo for Linux, and Linux VServer. As a representative instance of container based systems, this paper describes the design and implementation of Linux VServer. In addition, it contrasts the architecture of Linux VServer with current generations of Xen, and shows how Linux VServer provides comparable support for isolation and superior system efficiency.
EuroSys '07
Stephen Soltesz Herbert Potzl Marc E Fiuczynski Andy C Bavier Larry L Peterson
15954550
An updated performance comparison of virtual machines and Linux containers
Cloud computing makes extensive use of virtual machines because they permit workloads to be isolated from one another and for the resource usage to be somewhat controlled. In this paper, we explore the performance of traditional virtual machine (VM) deployments, and contrast them with the use of Linux containers. We use KVM as a representative hypervisor and Docker as a container manager. Our results show that containers result in equal or better performance than VMs in almost all cases. Both VMs and containers require tuning to support I/Ointensive applications. We also discuss the implications of our performance results for future cloud architectures.
2015 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)
Wes Felter Alexandre Ferreira Ramakrishnan Rajamony Juan C Rubio
1810240
High Performance Cloud Computing: A View of Scientific Applications
Scientific computing often requires the availability of a massive number of computers for performing large scale experiments. Traditionally, these needs have been addressed by using high performance computing solutions and installed facilities such as clusters and super computers, which are difficult to setup, maintain, and operate. Cloud computing provides scientists with a completely new model of utilizing the computing infrastructure. Compute resources, storage resources, as well as applications, can be dynamically provisioned (and integrated within the existing infrastructure) on a pay per use basis. These resources can be released when they are no more needed. Such services are often offered within the context of a Service Level Agreement (SLA) which ensure the desired Quality of Service (QoS) Aneka, an enterprise Cloud computing solution, harnesses the power of compute resources by relying on private and public Clouds and delivers to users the desired QoS. Its flexible and service based infrastructure supports multiple programming paradigms that make Aneka address a variety of different scenarios: from finance applications to computational science. As examples of scientific computing in the Cloud, we present a preliminary case study on using Aneka for the classification of gene expression data and the execution of fMRI brain imaging workflow.
2009 10th International Symposium on Pervasive Systems, Algorithms, and Networks
Christian Vecchiola Suraj Pandey Rajkumar Buyya
15050539
Performance Analysis of High Performance Computing Applications on the Amazon Web Services Cloud
Cloud computing has seen tremendous growth, particularly for commercial web applications. The on demand, pay as you go model creates a flexible and cost effective means to access compute resources. For these reasons, the scientific computing community has shown increasing interest in exploring cloud computing. However, the underlying implementation and performance of clouds are very different from those at traditional supercomputing centers. It is therefore critical to evaluate the performance of HPC applications in today's cloud environments to understand the tradeoffs inherent in migrating to the cloud. This work represents the most comprehensive evaluation to date comparing conventional HPC platforms to Amazon EC2, using real applications representative of the workload at a typical supercomputing center. Overall results indicate that EC2 is six times slower than a typical mid range Linux cluster, and twenty times slower than a modern HPC system. The interconnect on the EC2 cloud platform severely limits performance and causes significant variability.
2010 IEEE Second International Conference on Cloud Computing Technology and Science
Keith R Jackson Lavanya Ramakrishnan Krishna Muriki Shane Canon Shreyas Cholia John Shalf Harvey J Wasserman Nicholas J Wright
22345921
A GPGPU Transparent Virtualization Component for High Performance Computing Clouds
The GPU Virtualization Service (gVirtuS) presented in this work tries to fill the gap between in house hosted computing clusters, equipped with GPGPUs devices, and pay for use high performance virtual clusters deployed via public or private computing clouds. gVirtuS allows an instanced virtual machine to access GPGPUs in a transparent and hypervisor independent way, with an overhead slightly greater than a real machine/GPGPU setup. The performance of the components of gVirtuS is assessed through a suite of tests in different deployment scenarios, such as providing GPGPU power to cloud computing based HPC clusters and sharing remotely hosted GPGPUs among HPC nodes.
Euro Par
Giulio Giunta Raffaele Montella Giuseppe Agrillo Giuseppe Coviello
5136811
Scalable I/O forwarding framework for high performance computing systems
Current leadership class machines suffer from a significant imbalance between their computational power and their I/O bandwidth. While Moore's law ensures that the computational power of high performance computing systems increases with every generation, the same is not true for their I/O subsystems. The scalability challenges faced by existing parallel file systems with respect to the increasing number of clients, coupled with the minimalistic compute node kernels running on these machines, call for a new I/O paradigm to meet the requirements of data intensive scientific applications. I/O forwarding is a technique that attempts to bridge the increasing performance and scalability gap between the compute and I/O components of leadership class machines by shipping I/O calls from compute nodes to dedicated I/O nodes. The I/O nodes perform operations on behalf of the compute nodes and can reduce file system traffic by aggregating, rescheduling, and caching I/O requests. This paper presents an open, scalable I/O forwarding framework for high performance computing systems. We describe an I/O protocol and API for shipping function calls from compute nodes to I/O nodes, and we present a quantitative analysis of the overhead associated with I/O forwarding.
2009 IEEE International Conference on Cluster Computing and Workshops
Nawab Ali Philip H Carns Kamil Iskra Dries Kimpe Samuel Lang Robert Latham Robert B Ross Lee Ward P Sadayappan
15239264
Recommendations for Virtualization Technologies in High Performance Computing
The benefits of virtualization are typically considered to be server consolidation, (leading to the reduction of power and cooling costs) increased availability, isolation, ease of operating system deployment and simplified disaster recovery. High Performance Computing (HPC) environments pose one main challenge for virtualization: the need to maximize throughput with minimal loss of CPU and I/O efficiency. However, virtualization is usually evaluated in terms of enterprise workloads and assumes that servers are underutilized and can be consolidated. In this paper we evaluate the performance of several virtual machine technologies in the context of HPC. A fundamental requirement of current high performance workloads is that both CPU and I/O must be highly efficient for tasks such as MPI jobs. This work benchmarks two virtual machine monitors, Open VZ and KVM, specifically focusing on I/O throughput since CPU efficiency has been extensively studied [1] Open VZ offers near native I/O performance. Amazon's EC2 "ClusterCompute Node" product is also considered for comparative purposes and performs quite well. The EC2 "Cluster ComputeNode" product utilizes the Xen hyper visor in hvm mode and 10Gbit/s Ethernet for high throughput communication. Therefore, we also briefly studied Xen on our hardware platform (in hvmmode) to determine if there are still areas of improvement in KVM that allow EC2 to outperform KVM (with InfiniBand host channel adapters operating at 20 Gbit/s) in MPI benchmarks. We conclude that KVM's I/O performance is sub optimal, potentially due to memory management problems in the hyper visor. Amazon'sEC2 service is promising, although further investigation is necessary to understand the effects of network based storage on I/O throughput in compute nodes. Amazon's offering may be attractive for users searching for "InfiniBand like" performance without the upfront investment required to build an InfiniBand cluster or users wishing to dynamically expand their cluster during periods of high demand.
2010 IEEE Second International Conference on Cloud Computing Technology and Science
Nathan Regola Jean-Christophe Ducom
17773509
Performance Effective and Low Complexity Task Scheduling for Heterogeneous Computing
Efficient application scheduling is critical for achieving high performance in heterogeneous computing environments. The application scheduling problem has been shown to be NP complete in general cases as well as in several restricted cases. Because of its key importance, this problem has been extensively studied and various algorithms have been proposed in the literature which are mainly for systems with homogeneous processors. Although there are a few algorithms in the literature for heterogeneous processors, they usually require significantly high scheduling costs and they may not deliver good quality schedules with lower costs. In this paper, we present two novel scheduling algorithms for a bounded number of heterogeneous processors with an objective to simultaneously meet high performance and fast scheduling time, which are called the Heterogeneous Earliest Finish Time (HEFT) algorithm and the Critical Path on a Processor (CPOP) algorithm. The HEFT algorithm selects the task with the highest upward rank value at each step and assigns the selected task to the processor, which minimizes its earliest finish time with an insertion based approach. On the other hand, the CPOP algorithm uses the summation of upward and downward rank values for prioritizing tasks. Another difference is in the processor selection phase, which schedules the critical tasks onto the processor that minimizes the total execution time of the critical tasks. In order to provide a robust and unbiased comparison with the related work, a parametric graph generator was designed to generate weighted directed acyclic graphs with various characteristics. The comparison study, based on both randomly generated graphs and the graphs of some real applications, shows that our scheduling algorithms significantly surpass previous approaches in terms of both quality and cost of schedules, which are mainly presented with schedule length ratio, speedup, frequency of best results, and average scheduling time metrics.
IEEE Trans. Parallel Distributed Syst.
Haluk Topcuoglu Salim Hariri Minyou Wu
2055321
A Technique for High Performance Data Compression
Data stored on disks and tapes or transferred over communications links in commercial computer systems generally contains significant redundancy. A mechanism or procedure which recodes the data to lessen the redundancy could possibly double or triple the effective data densitites in stored or communicated data. Moreover, if compression is automatic, it can also aid in the rise of software development costs. A transparent compression mechanism could permit the use of "sloppy" data structures, in that empty space or sparse encoding of data would not greatly expand the use of storage space or transfer time; however that requires a good compression procedure. Several problems encountered when common compression methods are integrated into computer systems have prevented the widespread use of automatic data compression. For example (1) poor runtime execution speeds interfere in the attainment of very high data rates; (2) most compression techniques are not flexible enough to process different types of redundancy; (3) blocks of compressed data that have unpredictable lengths present storage space management problems. Each compression This article was written while Welch was employed at Sperry Research Center; he is now employed with Digital Equipment Corporation. 8 m, 2 /R4/OflAb l strategy poses a different set of these problems and, consequently the use of each strategy is restricted to applications where its inherent weaknesses present no critical problems. This article introduces a new compression algorithm that is based on principles not found in existing commercial methods. This algorithm avoids many of the problems associated with older methods in that it dynamically adapts to the redundancy characteristics of the data being compressed. An investigation into possible application of this algorithm yields insight into the compressibility of various types of data and serves to illustrate system problems inherent in using any compression scheme. For readers interested in simple but subtle procedures, some details of this algorithm and its implementations are also described. The focus throughout this article will be on transparent compression in which the computer programmer is not aware of the existence of compression except in system performance. This form of compression is "noiseless," the decompressed data is an exact replica of the input data, and the compression apparatus is given no special program information, such as data type or usage statistics. Transparency is perceived to be important because putting an extra burden on the application programmer would cause
Computer
Terry A Welch
15562514
Performance Evaluation of Container Based Virtualization for High Performance Computing Environments
The use of virtualization technologies in high performance computing (HPC) environments has traditionally been avoided due to their inherent performance overhead. However, with the rise of container based virtualization implementations, such as Linux VServer, OpenVZ and Linux Containers (LXC) it is possible to obtain a very low overhead leading to near native performance. In this work, we conducted a number of experiments in order to perform an in depth performance evaluation of container based virtualization for HPC. We also evaluated the trade off between performance and isolation in container based virtualization systems and compared them with Xen, which is a representative of the traditional hypervisor based virtualization systems used today.
2013 21st Euromicro International Conference on Parallel, Distributed, and Network Based Processing
Miguel G Xavier Marcelo Veiga Neves Fabio Diniz Rossi Tiago Coelho Ferreto Timoteo Lange Cesar A F De Rose