spark
11818928
Spark: Cluster Computing with Working Sets
MapReduce and its variants have been highly successful in implementing large scale data intensive applications on commodity clusters. However, most of these systems are built around an acyclic data flow model that is not suitable for other popular applications. This paper focuses on one such class of applications: those that reuse a working set of data across multiple parallel operations. This includes many iterative machine learning algorithms, as well as interactive data analysis tools. We propose a new framework called Spark that supports these applications while retaining the scalability and fault tolerance of MapReduce. To achieve these goals, Spark introduces an abstraction called resilient distributed datasets (RDDs) An RDD is a read only collection of objects partitioned across a set of machines that can be rebuilt if a partition is lost. Spark can outperform Hadoop by 10x in iterative machine learning jobs, and can be used to interactively query a 39 GB dataset with sub second response time.
HotCloud
Matei A Zaharia Mosharaf Chowdhury Michael J Franklin Scott Shenker Ion Stoica
7956687
MLlib: Machine Learning in Apache Spark
Apache Spark is a popular open source platform for large scale data processing that is well suited for iterative machine learning tasks. In this paper we present MLlib, Spark's open source distributed machine learning library. MLlib provides efficient functionality for a wide range of learning settings and includes several underlying statistical, optimization, and linear algebra primitives. Shipped with Spark, MLlib supports several languages and provides a high level API that leverages Spark's rich ecosystem to simplify the development of end to end machine learning pipelines. MLlib has experienced a rapid growth due to its vibrant open source community of over 140 contributors, and includes extensive documentation to support further growth and to let users quickly get up to speed.
J. Mach. Learn. Res.
Xiangrui Meng Joseph K Bradley Burak Yavuz Evan R Sparks Shivaram Venkataraman Davies Liu Jeremy Freeman D B Tsai Manish Amde Sean Owen Doris Xin Reynold Xin Michael J Franklin Reza Bosagh Zadeh Matei A Zaharia Ameet S Talwalkar
61154484
Apache Spark
Encyclopedia of Big Data Technologies
Alexandre da Silva Veith Marcos Dias de Assuncao
182838950
Preparation of mullite TiB2 CNTs hybrid composite through spark plasma sintering
Abstract A near fully dense mullite TiB2 CNTs hybrid composite was prepared successfully trough spark plasma sintering. 1 wt%CNT and 10 wt%TiB2 were mixed with nano sized mullite powders using a high energy mixer mill. Spark plasma sintering was carried out at 1350 degC under the primary and final pressure of 10 MPa and 30 MPa, respectively. XRD results showed mullite and TiB2 as dominant crystalline phases accompanied by tiny peaks of alumina. The microstructure of prepared composites demonstrated uniform distribution of TiB2 reinforcements in mullite matrix without any pores and porosities as a result of near fully densified spark plasma sintered composite. The fracture surface of composite revealed a proper bonding of TiB2 with mullite matrix and also areas with CNTs tunneling and superficies as a result of pulling out phenomenon. The flexural strength of 531 28 MPa, Vickers harness of 18.31 0.3 GPa, and fracture toughness of 5.46 0.12 MPa m 1/2 were achieved for prepared composites as the measured mechanical properties.
Ceramics International
Yasin Orooji Ehsan Ghasali Mostafa Moradi Mohammad Derakhshandeh Masoud Alizadeh Mehdi Shahedi Asl Touradj Ebadzadeh
2560213
Spark SQL: Relational Data Processing in Spark
Spark SQL is a new module in Apache Spark that integrates relational processing with Spark's functional programming API. Built on our experience with Shark, Spark SQL lets Spark programmers leverage the benefits of relational processing (e.g. declarative queries and optimized storage) and lets SQL users call complex analytics libraries in Spark (e.g. machine learning) Compared to previous systems, Spark SQL makes two main additions. First, it offers much tighter integration between relational and procedural processing, through a declarative DataFrame API that integrates with procedural Spark code. Second, it includes a highly extensible optimizer, Catalyst, built using features of the Scala programming language, that makes it easy to add composable rules, control code generation, and define extension points. Using Catalyst, we have built a variety of features (e.g. schema inference for JSON, machine learning types, and query federation to external databases) tailored for the complex needs of modern data analysis. We see Spark SQL as an evolution of both SQL on Spark and of Spark itself, offering richer APIs and optimizations while keeping the benefits of the Spark programming model.
SIGMOD Conference
Michael Armbrust Reynold Xin Cheng Lian Yin Huai Davies Liu Joseph K Bradley Xiangrui Meng Tomer Kaftan Michael J Franklin Ali Ghodsi Matei A Zaharia
207238620
Apache Spark
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applications.
Commun. ACM
Matei A Zaharia Reynold Xin Patrick Wendell Tathagata Das Michael Armbrust Ankur Dave Xiangrui Meng Josh Rosen Shivaram Venkataraman Michael J Franklin Ali Ghodsi Joseph E Gonzalez Scott Shenker Ion Stoica
53353858
Field Assisted Sintering Technology/Spark Plasma Sintering: Mechanisms, Materials, and Technology Developments
Field assisted sintering technology/Spark plasma sintering is a low voltage, direct current (DC) pulsed current activated, pressure assisted sintering, and synthesis technique, which has been widely applied for materials processing in the recent years. After a description of its working principles and historical background, mechanical, thermal, electrical effects in FAST/SPS are presented along with the role of atmosphere. A selection of successful materials development including refractory materials, nanocrystalline functional ceramics, graded, and non equilibrium materials is then discussed. Finally, technological aspects (advanced tool concepts, temperature measurement, finite element simulations) are covered.
Olivier Guillon Jesus Gonzalez-Julian Benjamin Dargatz T Kessel Gabi Schierning Jan Rathel Mathias Herrmann
2180634
Benchmarking Streaming Computation Engines: Storm, Flink and Spark Streaming
Streaming data processing has been gaining attention due to its application into a wide range of scenarios. To serve the booming demands of streaming data processing, many computation engines have been developed. However, there is still a lack of real world benchmarks that would be helpful when choosing the most appropriate platform for serving real time streaming needs. In order to address this problem, we developed a streaming benchmark for three representative computation engines: Flink, Storm and Spark Streaming. Instead of testing speed of light event processing, we construct a full data pipeline using Kafka and Redis in order to more closely mimic the real world production scenarios. Based on our experiments, we provide a performance comparison of the three data engines in terms of 99th percentile latency and throughput for various configurations.
2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)
Sanket Chintapalli Derek Dagit Bobby Evans Reza Farivar Thomas Graves Mark Holderbaugh Zhuo Liu Kyle Nusbaum Kishorkumar Patil Boyang Peng Paul Poulosky
99630372
Knocking combustion in spark ignition engines
Abstract Knocking combustion research is crucially important because it determines engine durability, fuel consumption, and power density, as well as noise and emission performance. Current spark ignition (SI) engines suffer from both conventional knock and super knock. Conventional knock limits raising the compression ratio to improve thermal efficiency due to end gas auto ignition, while super knock limits the desired boost to improve the power density of modern gasoline engines due to detonation. Conventional combustion has been widely studied for many years. Although the basic characteristics are clear, the correlation between the knock index and fuel chemistry, pressure oscillations and heat transfer, and auto ignition front propagation, are still in early stages of understanding. Super knock combustion in highly boosted spark ignition engines with random pre ignition events has been intensively studied in the past decade in both academia and industry. These works have mainly focused on the relationship between pre ignition and super knock, source analyses of pre ignition, and the effects of oil/fuel properties on super knock. The mechanism of super knock has been recently revealed in rapid compression machines (RCM) under engine like conditions. It was found that detonation can occur in modern internal combustion engines under high energy density conditions. Thermodynamic conditions and shock waves influence the combustion wave and detonation initiation modes. Three combustion wave modes in the end gas have been visualized as deflagration, sequential auto ignition and detonation. The most frequently observed detonation initiation mode is shock wave reflection induced detonation (SWRID) Compared to the effect of shock compression and negative temperature coefficient (NTC) combustion on ignition delay, shock wave reflection is the main cause of near wall auto ignition/detonation. Finally, suppression methods for conventional knock and super knock in SI engines are reviewed, including use of exhaust gas recirculation (EGR) the injection strategy, and the integration of a high tumble high EGR Atkinson/Miller cycle. This paper provides deep insights into the processes occurring during knocking combustion in spark ignition engines. Furthermore, knock control strategies and combustion wave modes are summarized, and future research directions, such as turbulence shock reaction interaction theory, detonation suppression and utilization, and super knock solutions, are also discussed.
Zhi Wang Hui Liu Rolf Reitz
11150229
GraphX: a resilient distributed graph system on Spark
From social networks to targeted advertising, big graphs capture the structure in data and are central to recent advances in machine learning and data mining. Unfortunately, directly applying existing data parallel tools to graph computation tasks can be cumbersome and inefficient. The need for intuitive, scalable tools for graph computation has lead to the development of new graph parallel systems (e.g. Pregel, PowerGraph) which are designed to efficiently execute graph algorithms. Unfortunately, these new graph parallel systems do not address the challenges of graph construction and transformation which are often just as problematic as the subsequent computation. Furthermore, existing graph parallel systems provide limited fault tolerance and support for interactive data mining. We introduce GraphX, which combines the advantages of both data parallel and graph parallel systems by efficiently expressing graph computation within the Spark data parallel framework. We leverage new ideas in distributed graph representation to efficiently distribute graphs as tabular data structures. Similarly, we leverage advances in data flow systems to exploit in memory computation and fault tolerance. We provide powerful new operations to simplify graph construction and transformation. Using these primitives we implement the PowerGraph and Pregel abstractions in less than 20 lines of code. Finally, by exploiting the Scala foundation of Spark, we enable users to interactively load, transform, and compute on massive graphs.
GRADES
Reynold Xin Joseph E Gonzalez Michael J Franklin Ion Stoica