Quickstart Big Data
Flebbe, Olaf
How to scale out an engineering workload in HPC with Apache Bigtop. The Speaker will present how to setup up Big Data scenarios using the toolscontained within the Apache Bigtop project. Apache Bigtop is a framework tocompile, package, test and deploy Big Data components supporting all majorLinux distros. The speaker will demonstrate how to install and configure asingle node or deploy to a cluster using the supplied package repositories andpuppet manifests. As an example usecase he will demonstrate how to run ApacheSpark to scale out a python engineering workload from single node to a HPCenvironment.
 Thank you for letting me into this room. It's very crowded outside. It's a great honor to be standing inside. So I will talk about a consulting job I did. First of all, some words about me. I did my PhD in computational physics a long time ago. I was involved in projects like Minix for 68k, the mathematics library for Linux. I did the high precision math. Pearl Python, porting to the Psyon 5, flight simulator, that's some contributions. And I'm a maintainer for the MSKT-UTIL. It's Active Directory Kerber's integration tool. Right now, for the moment, I am the PMC of the Apache Big Top project. More about that later. In my real life, the other world life, I'm a software architect for connected e-bikes. And the thing I was talking about is how to attack a problem when you have to do some consulting jobs and with a big data label on it. And the problem I had to solve was from Chem Informatics. You have for the development of medical drugs. So the problem at hand is you have a database, a large database of molecules that can be produced quite cheaply, for instance. And you have this medical database of things is written down, for instance, in the smallest notation. That's a complicated notation to the chemical structure of this thing here. I didn't remember the name of it. I just picked randomly one out of it. And the problem is to look for substructures into these molecules, which can make you healthy or something like that. So that's not a new problem. There's a commercial solution for it. You throw an enterprise database in it, and you have something like a cartridge in it. And you can make a SQL query and you get results. The problem is the duration is very long. For instance, it needs about a day or something. It's not very reliable, and it's very expensive. So the customer looked out for a big data approach to this problem. And fortunately, there's an open source project called RD Kit. It's a beautiful library for chemo-traumatics. And you can do something like that. You read in a molecule from smiles, and you put in the smiles notification and you have your molecule object, a Python object, and can simply say it has a substructure match from the other small notation. It gives you true or false. That's fine. Okay. So it's almost the same thing. We have the ingredients, a time-consuming job. We have a large data set and make it fast. So we have the environment we wanted to use. It's a big data cluster or an HPC cluster. We have both. So what can we do? So now about how not to scale out. You can simply benchmark it and see that the reading in the smiles notation and constructing the molecule object is the most time-consuming thing in this whole problem. And okay, we can simply read it in one time and dumping the, serializing the molecules. It's called pickling in Python. You can dump it to a file, as though you do not have to reconstruct all the molecules at any time. So that's a huge gain in runtime, but it does not scale anything. We are looking out for scale. So in order to make the program faster, stronger, more machines in it. So we are simply looking at the problem. It's simply an HP, an EP problem. It's embarrassingly parallel. So we can simply divide the database into small chunks and so each chunks at different machines. Quite easy. And the big data approach is to distribute the algorithm to the machines and not the data coming to one machine. So yes, the beautiful framework for us is Spark. It's a Spark core, no special ingredients needed. And we use the RDD paradigm that's resilient data distribution. I think it's called that is that the files are hacked into different components so we can distribute it to all the machines. And Spark is beautiful because it runs each on HPC and big data in the environment. It does not need, is not tied to the HDFS, for instance. So all I did is with the instructions at Spark. I never haven't used Spark before. And it's quite easy. We read in a text file. Okay, in fact I'm reading in the pickled file but forget about it. Text file, I have the input and I can distribute the algorithm. That's the algorithm here, the function to the cluster and do some aggregation in order to get benchmarking results. That's all. All I have to do is if I remember about constructing an MPI job and thinking what I have to do with all the initialization and it isn't failsafe. This is inherently failsafe because Spark handles if a node is crashing or had misbehaving or something like that. And only realize on that every node can read that file. So if you have ever have some kind of shared file system, I can distribute it and run it in the environment. So HPC mode is quite straightforward. Use a cluster file system, dump the Spark jar on it, distribute the data in the same directory and okay, it does not use the locality of data store but this is CPU bound, not data bound. And we can use the standalone mode of Spark. And okay, let's compare about with a big data setup. Okay, I have to make some advertisement for the Apache Big Top distribution and only one minute left. Okay. Okay, the Apache Big Top is the Debian of big data distributions. It's reused by Google, Cloudera, Canonical and UDPI. Please turn around, Roman. And it contains all the usual stuff and it's important to have a compiler environment. We have a repositories. We have provisioning with Docker compose for testing, of course. The August stack is broken now, sorry. We had deployment templates based on puppet orchestration. If one needs to like to have it, the Judo charms, that's the Canonical thing. We have an automatic testing environment and we have non-intel architectures. And that's a glimpse for CI Compile System. That's the distribution, the Linux distribution center is 7.6, Debian Fedora, Fedoria on PowerVC and Ubuntu on ARM64 and that's something like Hadoop, Giraffe, Hive and something like that. That's all these big data ecosystem. And simply deploying it with the puppet scripts, use the HDFS, put the data into it, use the yarn mode to spark and unpack the spark too because we do not support it until now and run. Okay, it works. Preview of BigTop 1.2 is we will have spark too but unfortunately it's not finished until now and we need more help. Please join us at BigTop Apache org. So let me conclude. The problem runs much better on the HPC environment because it's compute intensive, not data pipeline. That's simply the cause, the big data environment is optimized for throughput, data throughput, not for HPC runtime. And the problem scales really well, it's over N. But unfortunately, if we have, that's the number of machines we put in and we have some runtime, the total runtime I'm waiting for the job, that's the commercial solution because it's the fixed environment and our end is, and that one is too much for the customer to pay. So we will have to investigate, somehow someone has to investigate how well the, where we can speed up it much better. Thanks. Any questions?
 Thank you for letting me into this room. It's very crowded outside. It's a great honor to be  standing inside. So I will talk about a consulting job I did. First of all, some words about me.  I did my PhD in computational physics a long time ago. I was involved in projects like  Minix for 68k, the mathematics library for Linux. I did the high precision math. Pearl Python,  porting to the Psyon 5, flight simulator, that's some contributions. And I'm a maintainer for  the MSKT-UTIL. It's Active Directory Kerber's integration tool. Right now, for the moment,  I am the PMC of the Apache Big Top project. More about that later. In my real life, the  other world life, I'm a software architect for connected e-bikes. And the thing I was talking  about is how to attack a problem when you have to do some consulting jobs and with a big data  label on it. And the problem I had to solve was from Chem Informatics. You have for the  development of medical drugs. So the problem at hand is you have a database, a large database of  molecules that can be produced quite cheaply, for instance. And you have this medical database  of things is written down, for instance, in the smallest notation. That's a complicated notation  to the chemical structure of this thing here. I didn't remember the name of it. I just picked  randomly one out of it. And the problem is to look for substructures into these molecules,  which can make you healthy or something like that. So that's not a new problem. There's a  commercial solution for it. You throw an enterprise database in it, and you have something like a  cartridge in it. And you can make a SQL query and you get results. The problem is the duration  is very long. For instance, it needs about a day or something. It's not very reliable,  and it's very expensive. So the customer looked out for a big data approach to this problem.  And fortunately, there's an open source project called RD Kit. It's a beautiful library for  chemo-traumatics. And you can do something like that. You read in a molecule from smiles, and you  put in the smiles notification and you have your molecule object, a Python object, and can  simply say it has a substructure match from the other small notation. It gives you true or false.  That's fine. Okay. So it's almost the same thing. We have the ingredients, a time-consuming job. We  have a large data set and make it fast. So we have the environment we wanted to use. It's a big  data cluster or an HPC cluster. We have both. So what can we do? So now about how not to scale  out. You can simply benchmark it and see that the reading in the smiles notation and constructing  the molecule object is the most time-consuming thing in this whole problem. And okay, we can  simply read it in one time and dumping the, serializing the molecules. It's called pickling in  Python. You can dump it to a file, as though you do not have to reconstruct all the molecules  at any time. So that's a huge gain in runtime, but it does not scale anything. We are looking out  for scale. So in order to make the program faster, stronger, more machines in it. So we are simply  looking at the problem. It's simply an HP, an EP problem. It's embarrassingly parallel. So we can  simply divide the database into small chunks and so each chunks at different machines. Quite easy.  And the big data approach is to distribute the algorithm to the machines and not the data coming  to one machine. So yes, the beautiful framework for us is Spark. It's a Spark core, no special  ingredients needed. And we use the RDD paradigm that's resilient data distribution. I think it's  called that is that the files are hacked into different components so we can distribute it to  all the machines. And Spark is beautiful because it runs each on HPC and big data in the environment.  It does not need, is not tied to the HDFS, for instance. So all I did is with the instructions  at Spark. I never haven't used Spark before. And it's quite easy. We read in a text file. Okay,  in fact I'm reading in the pickled file but forget about it. Text file, I have the input and I can  distribute the algorithm. That's the algorithm here, the function to the cluster and do some  aggregation in order to get benchmarking results. That's all. All I have to do is if I remember  about constructing an MPI job and thinking what I have to do with all the initialization and it  isn't failsafe. This is inherently failsafe because Spark handles if a node is crashing or  had misbehaving or something like that. And only realize on that every node can read that file. So  if you have ever have some kind of shared file system, I can distribute it and run it in the  environment. So HPC mode is quite straightforward. Use a cluster file system, dump the Spark  jar on it, distribute the data in the same directory and okay, it does not use the locality of  data store but this is CPU bound, not data bound. And we can use the standalone mode of Spark.  And okay, let's compare about with a big data setup. Okay, I have to make some advertisement for  the Apache Big Top distribution and only one minute left. Okay. Okay, the Apache Big Top is  the Debian of big data distributions. It's reused by Google, Cloudera, Canonical and UDPI. Please  turn around, Roman. And it contains all the usual stuff and it's important to have a compiler  environment. We have a repositories. We have provisioning with Docker compose for testing,  of course. The August stack is broken now, sorry. We had deployment templates based on  puppet orchestration. If one needs to like to have it, the Judo charms, that's the Canonical  thing. We have an automatic testing environment and we have non-intel architectures. And that's  a glimpse for CI Compile System. That's the distribution, the Linux distribution center is 7.6,  Debian Fedora, Fedoria on PowerVC and Ubuntu on ARM64 and that's something like Hadoop,  Giraffe, Hive and something like that. That's all these big data ecosystem. And simply deploying  it with the puppet scripts, use the HDFS, put the data into it, use the yarn mode to spark and  unpack the spark too because we do not support it until now and run. Okay, it works. Preview of  BigTop 1.2 is we will have spark too but unfortunately it's not finished until now and we need more  help. Please join us at BigTop Apache org. So let me conclude. The problem runs much better on the  HPC environment because it's compute intensive, not data pipeline. That's simply the cause,  the big data environment is optimized for throughput, data throughput, not for HPC runtime. And the  problem scales really well, it's over N. But unfortunately, if we have, that's the number  of machines we put in and we have some runtime, the total runtime I'm waiting for the job,  that's the commercial solution because it's the fixed environment and our end is, and that one is  too much for the customer to pay. So we will have to investigate, somehow someone has to  investigate how well the, where we can speed up it much better. Thanks. Any questions?
Quickstart_Big_Data_0_131.png Quickstart_Big_Data_132_359.png Quickstart_Big_Data_360_1559.png Quickstart_Big_Data_1560_5171.png Quickstart_Big_Data_5172_7091.png Quickstart_Big_Data_7092_8771.png Quickstart_Big_Data_8772_11243.png Quickstart_Big_Data_11244_11591.png Quickstart_Big_Data_11592_12059.png Quickstart_Big_Data_12060_13955.png Quickstart_Big_Data_13956_14279.png Quickstart_Big_Data_14280_16163.png Quickstart_Big_Data_16164_16271.png Quickstart_Big_Data_16272_16288.png
Thank you for letting me into this room. It's very crowded outside. It's a great honor to be standing inside. So I will talk about a consulting job I did. First of all, some words about me. I did my PhD in computational physics a long time ago. I was involved in projects like Minix for 68k, the mathematics library for Linux. I did the high precision math. Pearl Python, porting to the Psyon 5, flight simulator, that's some contributions. And I'm a maintainer for the MSKT-UTIL. It's Active Directory Kerber's integration tool. Right now, for the moment, I am the PMC of the Apache Big Top project. More about that later. In my real life, the other world life, I'm a software architect for connected e-bikes. And the thing I was talking about is how to attack a problem when you have to do some consulting jobs and with a big data label on it. And the problem I had to solve was from Chem Informatics. You have for the development of medical drugs. So the problem at hand is you have a database, a large database of molecules that can be produced quite cheaply, for instance. And you have this medical database of things is written down, for instance, in the smallest notation. That's a complicated notation to the chemical structure of this thing here. I didn't remember the name of it. I just picked randomly one out of it. And the problem is to look for substructures into these molecules, which can make you healthy or something like that. So that's not a new problem. There's a commercial solution for it. You throw an enterprise database in it, and you have something like a cartridge in it. And you can make a SQL query and you get results. The problem is the duration is very long. For instance, it needs about a day or something. It's not very reliable, and it's very expensive. So the customer looked out for a big data approach to this problem. And fortunately, there's an open source project called RD Kit. It's a beautiful library for chemo-traumatics. And you can do something like that. You read in a molecule from smiles, and you put in the smiles notification and you have your molecule object, a Python object, and can simply say it has a substructure match from the other small notation. It gives you true or false. That's fine. Okay. So it's almost the same thing. We have the ingredients, a time-consuming job. We have a large data set and make it fast. So we have the environment we wanted to use. It's a big data cluster or an HPC cluster. We have both. So what can we do? So now about how not to scale out. You can simply benchmark it and see that the reading in the smiles notation and constructing the molecule object is the most time-consuming thing in this whole problem. And okay, we can simply read it in one time and dumping the, serializing the molecules. It's called pickling in Python. You can dump it to a file, as though you do not have to reconstruct all the molecules at any time. So that's a huge gain in runtime, but it does not scale anything. We are looking out for scale. So in order to make the program faster, stronger, more machines in it. So we are simply looking at the problem. It's simply an HP, an EP problem. It's embarrassingly parallel. So we can simply divide the database into small chunks and so each chunks at different machines. Quite easy. And the big data approach is to distribute the algorithm to the machines and not the data coming to one machine. So yes, the beautiful framework for us is Spark. It's a Spark core, no special ingredients needed. And we use the RDD paradigm that's resilient data distribution. I think it's called that is that the files are hacked into different components so we can distribute it to all the machines. And Spark is beautiful because it runs each on HPC and big data in the environment. It does not need, is not tied to the HDFS, for instance. So all I did is with the instructions at Spark. I never haven't used Spark before. And it's quite easy. We read in a text file. Okay, in fact I'm reading in the pickled file but forget about it. Text file, I have the input and I can distribute the algorithm. That's the algorithm here, the function to the cluster and do some aggregation in order to get benchmarking results. That's all. All I have to do is if I remember about constructing an MPI job and thinking what I have to do with all the initialization and it isn't failsafe. This is inherently failsafe because Spark handles if a node is crashing or had misbehaving or something like that. And only realize on that every node can read that file. So if you have ever have some kind of shared file system, I can distribute it and run it in the environment. So HPC mode is quite straightforward. Use a cluster file system, dump the Spark jar on it, distribute the data in the same directory and okay, it does not use the locality of data store but this is CPU bound, not data bound. And we can use the standalone mode of Spark. And okay, let's compare about with a big data setup. Okay, I have to make some advertisement for the Apache Big Top distribution and only one minute left. Okay. Okay, the Apache Big Top is the Debian of big data distributions. It's reused by Google, Cloudera, Canonical and UDPI. Please turn around, Roman. And it contains all the usual stuff and it's important to have a compiler environment. We have a repositories. We have provisioning with Docker compose for testing, of course. The August stack is broken now, sorry. We had deployment templates based on puppet orchestration. If one needs to like to have it, the Judo charms, that's the Canonical thing. We have an automatic testing environment and we have non-intel architectures. And that's a glimpse for CI Compile System. That's the distribution, the Linux distribution center is 7.6, Debian Fedora, Fedoria on PowerVC and Ubuntu on ARM64 and that's something like Hadoop, Giraffe, Hive and something like that. That's all these big data ecosystem. And simply deploying it with the puppet scripts, use the HDFS, put the data into it, use the yarn mode to spark and unpack the spark too because we do not support it until now and run. Okay, it works. Preview of BigTop 1.2 is we will have spark too but unfortunately it's not finished until now and we need more help. Please join us at BigTop Apache org. So let me conclude. The problem runs much better on the HPC environment because it's compute intensive, not data pipeline. That's simply the cause, the big data environment is optimized for throughput, data throughput, not for HPC runtime. And the problem scales really well, it's over N. But unfortunately, if we have, that's the number of machines we put in and we have some runtime, the total runtime I'm waiting for the job, that's the commercial solution because it's the fixed environment and our end is, and that one is too much for the customer to pay. So we will have to investigate, somehow someone has to investigate how well the, where we can speed up it much better. Thanks. Any questions?
the PMC of the Apache Big Top project. More about cluster or an HPC cluster. We have both. So what can we do? So now about how not to scale out. You different components so we can distribute it to and thinking what I have to do with all the and run it in the environment. So HPC mode is compiler environment. We have a repositories. We pay. So we will have to investigate, somehow