Migrating a Big Data Cluster from Linux to FreeBSD
Reuschling, Benedict
This talk will cover the on-going process of migrating a big data cluster from a Linux-only system to a mixed OS environment using more and more BSD. I'll cover how the cluster started, what it is used for and the current setup (hardware and software). Particular focus will be put on the migration of the fileserver that has been successfully migrated from a hardware RAID configuration using Linux to an OpenZFS-based FreeBSD setup. I'll detail how it was done, the planning, preparation, and important learnings I too away from the experience. The talk closes with an outline of what steps will follow in the future. The talk is intended for people interested in such setups, migration strategies, steps to take and pitfalls to avoid. I manage the Big Data cluster of the University of Applied Sciences, Darmstadt, Germany as system administrator. As such, I'm responsible for providing the compute resources to researchers and teachers during the semester, as well as students doing projects and thesis work on the cluster nodes. When I took over the cluster a couple of years ago, it was a Linux-only system. I've managed to convert more and more nodes to using FreeBSD and OpenZFS. My talk will focus on the how and why, elaborating benefits of the approach and rough edges that still need to be worked on. In the summer of 2018, I took on one of the biggest tasks yet: the migration of the central file server for the cluster. This fileserver provides the home directories to each node via NFS. I was successful in migrating it from a pure hardware RAID setup to a FreeBSD-based OpenZFS software RAID solution. This provides the usual benefits associated with OpenZFS such as compression, quotas and reservation, as well as data protection on various levels. My talk will detail the preparation that I took before the migration and provide some insight into how the setup works now. Future work is discussed and I hope to gather some feedback from the audience about topics like monitoring, backup, and outstanding work.
 So, hello. Thanks for coming to my talk here in the BSD Dev Room. Some people might remember last year speaking, me speaking about my engagement in speaking or in teaching BSD in my university and this is the actual sysadmin part of my role there. So, it's a nice combination because I can teach what I learned as a sysadmin and vice versa, I can teach the students a little bit what's involved. So, today I'm talking about migrating a big data cluster from Linux to free BSD slowly but steadily. So, this is going on for quite a while now and I want to give kind of a status report and also a little bit how I approach the whole thing. This is what I'm trying to cover in the 40 minutes that I was given and don't worry it's a little bit more technical and also a couple of social aspects are in there. So, why this talk? Sometimes we get at the foundation or at the free BSD table downstairs the question of yeah, I like BSD, I want to use it not only at home but also in my workplace, wherever that is but I'm the only person using BSD there. I'm surrounded by other people who use other operating systems, what can I do? And so, I thought well this is actually what I'm experiencing or have been experiencing. So, I thought let's create a talk about this and this is an experience report basically how I approached it and maybe you can get some ideas how to implement some of this in your own environments because we want to see BSD rising in the not only in the private space but also in the environments where you work or in the work spaces. So, as I wrote here there's no one true approach so I have no master recipe to give you but I have my story to give you which is not also a successful one but also fraught a little bit with peril so it's not the perfect approach but at least it's one way to get slowly but steadily a little bit better. So, the first thing that you do is look around and learn because if you're starting in a new company or you're with a company for a while then you make a certain, you look around and see certain things that you might like or certain things that you don't like and maybe you figure out okay this is where BSD would come in handy in this area that is basically making a list of things that you want to change. So, I use it also to describe to you what the environment is that I work in and yeah so this is the first slide here so one day I inherited a big data cluster at work. So, I started there and a colleague of mine had actually started this big data cluster project that they started from the need that the students wanted to do long-term measurements of performance like for no-escalar database, benchmarking or thesis work or just projects that they were doing and so they were granted a couple of machines as called big data clusters it's basically a rack of machines I have a picture in the next slide and so they built it up I wasn't involved in that part so I basically had to take what they left me or what they gave to me and so I took over this this is the big data cluster it's called and it's using it's used in the University of Applied Sciences in Darmstadt Germany where I work in the computer science department although nowadays the whole university can use this cluster and the person that initially administered was a Linux person oh how new is that and was using Ubuntu at home so you thought it would be a good idea to use Ubuntu that he knew in this cluster as well and so this might be one of the first mistakes that were made but I'm getting to that and so this is an old Linux environment so actually the whole university is mostly Linux or the computer science department couple of people are now getting more into the BSD's now that I did a little bit of advocacy work but basically back then this is a couple years back there was no BSD at all maybe me running this as virtual boxes or teaching students to use BSD in my classes but that's pretty much it so the US image that they were using was based on the actual computer lab machine image which is kind of weird because this is a server environment and lab machines are typically sitting there waiting for students to do some work on them so lab machines are typically more desktop oriented and the image is also desktop oriented I don't know why they choose to use that on the big data cluster but I'm getting to that later on and then the classic thing operating system on the first disk and the next three disks that are in each of those nodes are used for the NoSQL database data files and they also bought a central file server for exporting the home directories for the students and the professors via NFS-3 so this is fairly straightforward and fairly easy to use so here's the little for the graphics or for the hardware geeks here out there so this is what we use so we have old nodes these are the ones that I started with so it this model the C6220s aren't even on sale by Dell anymore because they went out of support a couple of years ago and we're now using the next generation models the C6320 and so it's basically 64 gig RAM machines 8 CPUs and 4 hard disks that's pretty much it what we're working with and so some of the newer nodes have a bit more memory and a couple of newer modern processors but that's pretty much the whole the only difference and the file server is a little bit less in the memory area which is kind of bad if you're thinking about using ZFS later but pretty much the CPUs are there with enough power and they have enough disk space to cover the computing needs or the NoSQL database needs that we have so the software in use when I took it over was basically a dub installation and they were using also pig which is kind of accelerating the hadoop queries a bit and couch base was in use MongoDB and nowadays we're also having a spark installation which has a couple of other components in there so this is basically all in the area of big data and data analytics and we have a study program now with the mathematics department and that's called data science so the student can take that four year or four semester course and become data scientists and they're using the cluster basically for lab work thesis work or project in this area with companies for example and so then I looked at okay let's look around what actually is supported on free BSD so it's the dub support on free BSD answer is yes and I wrote about this in the free BSD journal a couple of months ago how I use free BSD and how you can deploy free BSD or hadoop on free BSD with ZFS because the ZFS part makes it actually interesting couch base itself has no native BSD client there's only lip couch base in ports and there's nothing to download on the manufacturers or the vendors website so that needs a little bit of porting help I see a couple of port is in the back maybe that will be a thing to work on MongoDB there's a part available but it's not the latest version so currently there's four of zero five and in ports this version three point six and of course the researchers and the professors in my university are always interested in using the latest versions not that they want to use it every semester the newest version because they usually also want to make sure that their slides are staying valid for a couple of semesters so but in generally if you are writing research papers you always want to test with the latest versions of the software because otherwise they will not accept your paper or laugh at you the spark server itself we started that last year and that's kind of involved the actual playbook that I wrote to deploy this has actually over 180 steps and you can imagine how complicated it is because spark also includes the hadoop installation and so there's multiple dependencies there to you know make it all work okay so how do we start basically or what's the environment that I'm also working on so education is an interesting place to work in in that there's no production environment or the production is an environment is less important you would say than in a like a commercial environment where you have to have 24 seven uptime and all these things so my production environment usually only lasts like one semester which is like three to four months and then I basically can reinstall again because that's all the software being used in the next semester so I only have to take care of the software for like three months and then let's reinstall the whole thing again that I made so much work on to make it running and so lectures as I told you they tend to stay with all the software versions a little bit longer unless they're doing research in that case they need the latest and greatest versions of the no SQL software and the production as again is only the time frame when the students doing their labs which is typically five to six times and then next the other week is the other group so we have typically two lab groups and the cluster makes it actually easy to not be bound to a certain lab because they can use SSH to connect to these machines from home or wherever they are and don't have to be at a certain lab in a certain hour and can only do it for like how an hour and a half before the next class starts that makes the cluster relatively flexible in its use and the popularity since we have that introduced it's very popular because students see that they can run a lot of beefy benchmarks on that or make a lot more CPUs spin than they have at home and so the problem was when we talk about installation life cycles and updates that we also have to make sure that the operating system supports the no SQL databases and so for the longest time when they when you were still running on the Linux side of things they were running on the Ubuntu 14.4 version and we couldn't get to Ubuntu 16 or 18 today because for the longest time Hadoop had a problem with Ubuntu 16 you couldn't run a Hadoop cluster on Ubuntu 16 that was the vendor trying to get it working and we were just waiting and nothing happened and so the students kept asking hey can I upgrade this thing I want to use the newer version if you want to and I said well if you want to use Hadoop there's no chance you can try it and you would help me actually make it work but several students tried and never got there and then someday Hadoop folks got it working I don't know what was the showstopper there but then we could update to the latest version so on previously no problem latest versions perfectly supported no problems but other software was kind of difficult in that regard because it was always just available for Linux and BST is not even a download option on their website so that's a general problem and I also give away these lab machines or these cluster machines to students to do their thesis work on and it's always okay do you student what do you need as an operating system well I use Ubuntu whatever at home and please install this on the cluster okay sometimes I tried to install free BST for them and then ask them or gave them access to the note just to get an email five minutes later oh I don't know this operating system can you install Linux for me okay but a couple people tried and they actually did their thesis on the on those machines and yeah even on free BST because what they're mostly running are applications they don't care what operating systems are running on them and that way I can introduce free BST a little bit on the sneaky side and some people just want to run their databases and don't care what kind of operating system is running below okay so that's what the environment is that I'm working on or what I've been trying to make changes in and so the changes have been coming slowly but steadily so the rational here is I don't want to just introduce free BST because I like it I want to saw a couple of things where free BST could actually improve things or make things better and that's why I try to bring free BST in this environment and create something valuable for people so that I don't maybe they don't see the features or I've never heard about the free BST features and that's why I try to introduce this and told them about the cool features for example that ZFS provides or that Ubuntu or that Hadoop is running quicker on ZFS than on Ubuntu with a traditional EXT file system and so the thing that I told you about at the beginning that we were using the actual lab image for the cluster which is more as a server environment that includes we get the whole desktop craft from the labs into that environment so I had printer drivers on a server I had graphics drivers sorry Nicholas on a server that will never run anything graphical because you need the CPU power to do the actual benchmarking and so I took that image and actually was cutting and slashing stuff out of it which I don't need and that actually reduced the install time of that image from 25 minutes to 11.5 roughly and I think I can cut this down even further removing more Linux stuff that we don't run on the cluster anyway and the free BST install itself just takes three minutes probably less if I can tune it a little bit more because the only thing that I need is basically get the OS running partitioning and ZFS setup and all that and set up the networking and maybe some LDAP so that the students can log in with their student ID and that's pretty much it then I reboot I don't have to install software in this part I can install software later using Ansible or other software distribution or just package install that's pretty much it I don't need this as a full big image that I have to copy over the networking to the to the node which actually creates already some value because install times are going down the nodes are available faster again if a student is finished and the next student is already waiting for that node to become available I can say sure in 25 minutes or now in like 11.5 you can have the next node available because that's the install time that it takes and if I'm testing things and I need to reboot the machine because I totally messed it up and then I have quicker test cycles this way so first steps if you're in a new environment and want to try out free BSD and maybe it's a hardware or hardware environment it never worked on is the question of course is your favorite BSD working on this it doesn't have to be free BSD could be other BSDs as well but I want to use free BSD and so after the first boot I was kind of encouraged because a lot of devices were detected and because free BSD is the actual server that's where free BSD is coming from it's a server desktop or server desktop server operating system and there's a lot of drivers available for you know these bigger and more expensive hardware parts and then the question is well I can install it but that's the machine actually reboot into the new operating system or is it just forgetting about bootloaders and all the other things that are required and luckily it did so then I knew that this machine could run free BSD then what I did is was quickly start and a process to automate the whole in OS installation because as much as you like your favorite operating system there's nothing more boring than just installing it all over again it's clicking the same prompts entering the same information all over again and so that's the first thing that I did trying to create this little ISO image that's completely automated just load it and let the machine boot from that and it automatically installs and you just have to watch and drink coffee and so also network setup is a bit interesting in the university environment we do a little bit of IPv6 there we also have a couple of things where we try to separate networks from the students so that they don't break into I don't know our central servers and change their grades or whatever so these are the there's a separate network for the big data clusters and it's also a bit secure this way and we have an LDAP server as I mentioned that is providing the whole accounts for the department and for the students so that the students don't have to create their own accounts and remember their passwords all the time so it's just one password they have to remember and of course the NFS setup from the central file server to mount the home directories for the students and that is imported from from them and then they have every file that they have in the home directory on each note that they have access to and of course I could reuse all of the many of the things from the Linux desktop environment that they provided initially but I just had to move that to BSD where paths are different or where some Linux specific things that don't work on BSD's and that's the actual first steps that I took to create a note that is complete free BSD note and we're still not talking about the application showing it this is just operating system and networking okay so then we need to tackle bigger problems because then we have already established some kind of foothold on the on the cluster with the BSD you know that it's running and then the question becomes is the actual software that the professors are using running on that environment and again this I get a lot the first sentence here well actually we don't care where so this is the database group where the cluster is located so and they pretty much don't care what operating system they're running on because they're mostly focusing on the database side and don't care what operating system is running below and as long as their favorite noise grid databases are working then they can just do their benchmarks and whatever they need for their research purposes and so the question is why are you not just running everything in beehive because then you can have a bit more flexibility and just providing Linux machines that are actually running on a free BSD environment and you just fake that there is a Linux distribution in there and they are just running their same environment that they are used to and that could be done the problem is that the professors actually or the students that are working on that are actually doing performance benchmarking and creating papers and thesis work and as much as I like beehive and its performance or its thinness in the layer between the operating system and the virtualization there's still a little bit of overhead and that little overhead can be seen in benchmarks or could be seen in benchmarks and that might skew the results a little bit so it's kind of difficult to say okay let's fake everything in beehive overall we had used beehive a couple of times because as I said the cluster was becoming very very popular and a lot of users wanted to use the cluster and I said oh sorry I'm out of notes I don't have any notes left we have to wait until the next student is finished with his thesis or her thesis and so what I tried to do is give everyone a little bit of space on the cluster in a little beehive because they don't usually run everything on all notes or only need like four CPUs instead of eight or only 16 gigs of RAM instead of 32 so I could try to partition this a little bit and that worked a little bit when we were in the situation where a lot of students wanted to do the data science labs and we didn't have enough notes left so I had to create a couple of beehives to make that a little bit better to allow more users because we want to let the users use our node or our cluster as much as possible okay so that's the research part why we cannot do the beehive thing the other thing is you saw that there is there are four disks in there and it's immediately clear that everyone says well of course I created a rate Z rate 10 on that and perfect over four disks that's a good performance boost you also would get ZFS compression and all the other things to manage your IO better and all the cool things that Alan Jude could probably talk about and and of course you would also use a separate data set for each node SQL database it's much more flexible you have much more ways to manage your storage this way problem is same argument when they do scientific measurements or writing papers apparently if there's a rate layer involved somehow then the papers that you are submitting are flat out rejected the paper committees just look for and not just looking for that but if there's a rate layer somewhere if it's a hardware rate or a software rate then they just say well it's all all the benchmarking results are the result of the rate layer and not of your tuning mechanisms in the database layer and that was kind of a bummer for me because I really want to introduce ZFS into this environment because I think the professors would also like that and but when they have to do this paperwork or this research paper things and are rejected from conferences we didn't have that yet but that's the the downside of this if they wouldn't do any benchmarks then it's perfect it's the environment we're probably going to use and then I always thinking well if the paper committees reject those papers then you know how realistic or production ready is this kind of setup because then you could say well if you cannot do a rate then you just have to use a single disk but that single disk will fail I mean the disk in our cluster are like five years old now so I just wait every hour to get a result from the smart CTL to tell me that the disk has died but is this actually I mean this might be good for research but is this production things that we're talking about here so I'm kind of a little bit on the edge about that okay so next thing yeah the comment was here that the comparison is difficult if you're different you have different rate setups and different rate levels for comparison okay so now we have a free BSD a couple of free BSD machines in there next to the Linux machines that are already running the old kind of hardware and the old kind of software stack so now let's grow and extend a little bit so the biggest issue here is the file server oh we don't only have notes we also have a file server and since the file server has been serving these home directories to everyone and the research data set I mean we have a 370 gigabyte JSON file to load into the no SQL databases to I mean it compresses super well on ZFS or on I mean G-SIP but they don't have to they always have to ship the 300 gigabytes JSON file from one machine to the next to load this into the no SQL databases and then do some kind of schema management or I have no idea what they're doing but it's I mean 370 gigabytes of JSON okay and the file server is basically providing all these data sets and yeah welcome to the new world so since the file server has been serving all these files all the time there was no time to actually make it an upgrade of the software and to the actual operating system bits and again the file server wasn't set up by me and what they did they basically did a BIOS rate five so this is a hardware rate five and oh my god I wanted to run smart CTL on this thing and it gave me nothing because it couldn't look into individual disks made up of that rate five so I had no idea whether this disk were all dying or whether they are healthy so I did only see this one big blob of storage that they were providing down from the BIOS to the operating system and there was one issue from the start there were always so you would create a file on your home directory or on the file server shared and it will always have strange permissions you would get permissions like suddenly you had permissions that the dean was actually owner of that file or that the secretary was the group of this directory and I was like where is this coming from I mean the all the data is provided by the NFS or the L-LAP servers the L-LAP has all the group information and the group sizes or the group IDs why are they so wrong when it just touched a new file and so I after a while of looking at this I traced the problem down to the etc at user conf on linux previously don't have doesn't have that file maybe that's why I didn't find it right away and in there you have a value called first UID and if you're on an L-LAP setup you need to increase this a little bit so that the local accounts are distinguished from the ones provided from the L-LAP server and my predecessor didn't know about this or at least didn't do that because the the timestamp on the etc at user dot com was from the date when the server was installed no changes since then and once I bumped this all the files that were created had the correct ownership and permissions but you can imagine this was like five years into the big data cluster is live I'm not going into each individual's home directory and changing permissions that way so I thought okay let's this is a failed experiment let's reinstall the whole file server otherwise we're just coming into problems again and again I didn't like the hardware rate so I always wanted to use ZFS for this so I made a move into reinstalling the file server oh here's the first picture by the way whoo yeah so the process of migrating the file server was actually quite easy so I said okay one of the nodes is now becoming the new file server because I couldn't reinstall the file server in place even though we had relatively large semester breaks so where this could be possible but even during the semester breaks there were students writing their thesis on them or professors were doing projects on that so it always had to be something going on in the file server so I said okay now I take a note from the cluster and I install it the way I like it like ZFS and test all the things that I wanted to have that the later file server were going to was going to become and I copy all the data files over and create the mount points for the students and then let this run for a whole semester so this file server was actually offline no one was accessing it and they were just making all the calls to one of those nodes which was now the new file server and then during that time there were some issues or some things where students were confused where the file suddenly ended up but all these things went into basically a little file where I wrote down things that I needed to change or to make adjustments for the actual file server installation and once the semester was over which was pretty much painless so far I mean there were a couple of things that those were easy to fix and encouraged by those things I did the actual change to actually now say okay a big announcement to the whole department or the whole users from the big data cluster next week we installed the file server and surprisingly no one else was complaining because they were actually using this one note that I dedicated for as the new file server or the interim file server and they said yeah okay I'll give you one day and if everyone is complaining then well we still have the old data on the this one note that I used for it and so one day me very excited going into the server room getting in front of the file server and removing the old Ubuntu installation and booting 3bsd for the first time on it in which cause I removed the isn't it the nice the nicest thing to just go to a server and remove the hardware rate without having to worry about the data on it so that was cool so what I did is I created individual for each disk we have 24 of those in the file server so for each individual disk I created a rate zero because you couldn't do a no op export into the operating systems I had to so each disk has now is in a rate zero configuration exporting itself to basically the operating system and ZFS will basically pick that up and create the pool out of it and the data sets on top and ZFS installation went pretty well and in its cause I discovered the amazing MFI util so that's basically managing all the things on the operating system side that you would only do from the server BIOS and there are utilities like check each individual disks or run periodic checks and that's a very powerful tool and of course then after I was happy with the configuration in the file server setup that I wanted to have I copied all the data back from the node that was currently serving all the files and then said well thank you dear department now we have a ZFS based file server that I immediately saw the compression ratios and I was like yes so much disk space was saved and yeah everyone was interested in that and I told them about of course the other features that ZFS has quotas and reservation and all the other things because that's actually the interesting part then the students don't overwrite their own home directories anymore and I can limit the students to only say like two gigabytes of disk space that they're allowed to use maybe maybe four if I have good day but yeah that's pretty much the actual migration I'm pretty happy that I could make it work because the operating system on the file server is really getting old and I was worried that someday someone might be hacking that thing so I'm not actually too worried about external attackers I'm more worried about the students in computer science trying to do oh what's this utility doing and oh now I have a cluster machine on my control yeah not gonna looking good and then this is our new setup so you can see that I now export the NFS I also did an upgrade to NFS for because now I can serve it from ZFS and that way I can see what kind of data the students are actually saving in the home directory and also how much disk space it takes and all ZFS automatically compresses all that so we have like home per slash home is the home directories for the professors and the employees like me and they're separate from the students home directories in case someone else has the same name as the professors and they have their same home directory interesting concept and we have of course home data sets that has all these crazy 300 gigabyte JSON files and other things that are needed for the data science environment and yes no wrong permissions anymore because ZFS is understanding everything that the LDAP file server or the LDAP server is telling them about groups and permissions and I'm super happy now about this file server setup but there's more there was no monitoring when I started this big data cluster administration or the app there was previously a monitoring but it broke somehow and they didn't have the time to fix that so now we have a since the file server is now monitoring each individual node and to make sure that the file server who's running the monitoring isn't going down I have a little raspberry pi running free BSD on ZFS that's checking whether the file server is still there you know watches the watchers and so I get performance data from these nodes and if a student is making some crazy changes like running a Bitcoin miner which has happened once that's probably my talk next year yeah then I get the performance data and I see if a node goes down unexpectedly or if the network is having problems so that's kind of nice I'm using Isinga for that it's not it's not Nagios but I like Isinga the things that I saw and I will probably give a tutorial about Isinga in one of the future BSD conferences and it's a nice thing because I get a little bit more email this time every morning because each server is sending me information about you know how healthy the disks are and what kind of processes were running and how many students have tried to log into this one of course few futely and yeah this is the actual file server setup that we have and the monitoring that I added to it and again it's all now we have a free BSD file server we have a couple of free BSD nodes like 50 percent of the nodes are now free BSD and the actual monitoring is redundant with another Raspberry Pi node that is just checking whether the file server is still online and so in summary this is where we came from on the right side we had a desktop image based node installation with coupon to 14.4 with reinstalled times of 25 minutes we had NFS free mounts with some crazy ownership and permission problems we now had they had back then a little bit of Ansible scripting like create a user on the file server or create a home directory oh not not so not so much but it was at least a start and they had a broken monitoring back then and now with the free BSD setup we have a fully scripted server-centric installation we have much lower installation times this is just getting a coffee on free BSD and you want to take roughly 12 minutes I guess I can shave a little bit more on that but I have to see whether the newer versions which are probably now with more software that no one needs on a cluster we also have a ZFS based NFS file sharing server which provides ACLs and all other things that are interesting in the NFS4 world I'm playing a little bit with parallel NFS because that's been added to free BSD recently because that might be giving us a bit of performance boost and maybe a couple of redundant files over in case the file server might go down one day and I have pretty much everything Ansible scripted so all the Hadoop installations are there the MongoDBs the couch bases that we're using on the nodes I just have to say please install this on X machines and it gets installed and a couple of maintenance scripts like delete all the users from last semester and you know get a new boot environment installed and or snapshot done and things like that and most importantly we have a working and redundant monitoring in the learning system that keeps me awake at night telling me oh this one machine just went unexpectedly down and it's probably the disk that is five years old yeah so in retrospect it was a little bit sneaky where I introduced free BSD so I didn't actually say hey tomorrow I reinstalled the whole machine with free BSDs because that wouldn't work I would have gotten a lot of pushback from the professors who were used to their working environment and didn't want to have to suddenly deal with a new operating system that they've never heard of or I've never used and so you have to be a little bit sneaky at times and also patient I mean this is a process that's been going on for like the last three years and each semester I had a little bit more and more free BSD machines in there and as long as students or users are only using applications most of them are just oblivious they don't care what operating system they're running on there might be difference environment different environments and this is a very unique environment as a university but I was quite happy and this is not the end result the process is still going on as I had to experiment with other noise-cured databases on free BSD and seeing what kind of versions are available in ports. I collected a couple of tips and learnings at the end so as I said start small and incremental do don't do the whole exploration step into different environments that you are not familiar with start slow make incremental improvements and don't do a big bang thing of saying okay next week we're all running on a new operating system surprise and also let people give people the time to experiment with their operating system that they're now being given or their new way that their home directories are provided for example and let them just get a little bit of experience with that and once they have no complaints then you can just continue with the next step. Always testing is a big thing test everything before you roll it out so if you're not sure whether things work test again test again test different scenarios so for example for the NFS mounts I not only had to test whether the mount works on free BSD but also working on the legacy Ubuntu systems that we're using and there were some interesting things coming up so that also needed to be tested and write everything down if you're a sysadmin you're probably used to that so making notes about every config file change everything that you're just doing on the command line paths that might be different from your other operating systems any settings that you might do in your loader.con for rc.conf all these things might become important when you're doing a new server installation and back everything up oh my god if I would have lost all these datasets that are used for research I would have not my head again so if I lose some of the data that during the process of migrating the file servers that would not go well with the professor so that that way I backed up everything to multiple nodes to just be sure if one node would die I still had the data from that but I would encourage you to experiment and as long as I mean we have boot environments we have all kinds of mechanisms to save our necks and you never know what comes out of that I mean if I wouldn't have installed the file server I would have never in discovered the MFI utility and what kind of cool things it provides without going into the BIOS to make changes to any kind of settings for the rate controllers and again don't just do things because you like that operating system of yours make sure that you provide some kind of value to your users be that they don't have to buy the server hard disk in the next couple of months because you saved a lot of disk space using zfs compression or that they are now having the proper files and permissions that they are used to small things like that can make a lot of people smile and your users are happy that you are such a good sysadmin taking care of these systems sometimes you have to be willing to compromise like the systems that I'm running is now doesn't have the zfs setup that I'm actually initially permitted provided because that was the research and benchmarking things for the papers that wasn't going to happen but it's the environment that I'm working with and I hope that if we don't do any projects with benchmarking then we're not going to run this other setup and the rest is just common sense really and my time is up and I guess you have a couple questions thank you we have a couple of things about the cluster here if you're interested in a hardware and some of the environment then find that link thank you
 So, hello. Thanks for coming to my talk here in the BSD Dev Room. Some people might remember  last year speaking, me speaking about my engagement in speaking or in teaching BSD in my university  and this is the actual sysadmin part of my role there. So, it's a nice combination because  I can teach what I learned as a sysadmin and vice versa, I can teach the students a little bit  what's involved. So, today I'm talking about migrating a big data cluster from Linux to free BSD  slowly but steadily. So, this is going on for quite a while now and I want to give kind of a status  report and also a little bit how I approach the whole thing. This is what I'm trying to cover in  the 40 minutes that I was given and don't worry it's a little bit more technical and also a couple  of social aspects are in there. So, why this talk? Sometimes we get at the foundation or at the  free BSD table downstairs the question of yeah, I like BSD, I want to use it not only at home but  also in my workplace, wherever that is but I'm the only person using BSD there. I'm surrounded by  other people who use other operating systems, what can I do? And so, I thought well this is  actually what I'm experiencing or have been experiencing. So, I thought let's create a  talk about this and this is an experience report basically how I approached it and maybe you can  get some ideas how to implement some of this in your own environments because we want to see  BSD rising in the not only in the private space but also in the environments where you work or in  the work spaces. So, as I wrote here there's no one true approach so I have no master recipe to  give you but I have my story to give you which is not also a successful one but also fraught a  little bit with peril so it's not the perfect approach but at least it's one way to get slowly  but steadily a little bit better. So, the first thing that you do is look around and learn because  if you're starting in a new company or you're with a company for a while then you make a certain,  you look around and see certain things that you might like or certain things that you don't like  and maybe you figure out okay this is where BSD would come in handy in this area that is basically  making a list of things that you want to change. So, I use it also to describe to you what the  environment is that I work in and yeah so this is the first slide here so one day I inherited a  big data cluster at work. So, I started there and a colleague of mine had actually started this big  data cluster project that they started from the need that the students wanted to do long-term  measurements of performance like for no-escalar database, benchmarking or thesis work or just  projects that they were doing and so they were granted a couple of machines as called big data  clusters it's basically a rack of machines I have a picture in the next slide and so they  built it up I wasn't involved in that part so I basically had to take what they left me or what  they gave to me and so I took over this this is the big data cluster it's called and it's using  it's used in the University of Applied Sciences in Darmstadt Germany where I work in the computer  science department although nowadays the whole university can use this cluster and the person  that initially administered was a Linux person oh how new is that and was using Ubuntu at home so  you thought it would be a good idea to use Ubuntu that he knew in this cluster as well and so this  might be one of the first mistakes that were made but I'm getting to that and so this is an old  Linux environment so actually the whole university is mostly Linux or the computer science department  couple of people are now getting more into the BSD's now that I did a little bit of advocacy work  but basically back then this is a couple years back there was no BSD at all maybe me running this  as virtual boxes or teaching students to use BSD in my classes but that's pretty much it so the  US image that they were using was based on the actual computer lab machine image which is kind of  weird because this is a server environment and lab machines are typically sitting there waiting  for students to do some work on them so lab machines are typically more desktop oriented and the  image is also desktop oriented I don't know why they choose to use that on the big data cluster  but I'm getting to that later on and then the classic thing operating system on the first  disk and the next three disks that are in each of those nodes are used for the NoSQL database  data files and they also bought a central file server for exporting the home directories for  the students and the professors via NFS-3 so this is fairly straightforward and fairly easy to use  so here's the little for the graphics or for the hardware geeks here out there so this is what  we use so we have old nodes these are the ones that I started with so it this model the C6220s  aren't even on sale by Dell anymore because they went out of support a couple of years ago and we're  now using the next generation models the C6320 and so it's basically 64 gig RAM machines 8 CPUs  and 4 hard disks that's pretty much it what we're working with and so some of the newer nodes have  a bit more memory and a couple of newer modern processors but that's pretty much the whole the  only difference and the file server is a little bit less in the memory area which is kind of bad  if you're thinking about using ZFS later but pretty much the CPUs are there with enough power and  they have enough disk space to cover the computing needs or the NoSQL database needs that we have  so the software in use when I took it over was basically a dub installation and they were using  also pig which is kind of accelerating the hadoop queries a bit and couch base was in use MongoDB  and nowadays we're also having a spark installation which has a couple of other components in there  so this is basically all in the area of big data and data analytics and we have a study program now  with the mathematics department and that's called data science so the student can take that four  year or four semester course and become data scientists and they're using the cluster basically  for lab work thesis work or project in this area with companies for example and so then I looked at  okay let's look around what actually is supported on free BSD so it's the dub support on free BSD  answer is yes and I wrote about this in the free BSD journal a couple of months ago how I use free  BSD and how you can deploy free BSD or hadoop on free BSD with ZFS because the ZFS part makes it  actually interesting couch base itself has no native BSD client there's only lip couch base in  ports and there's nothing to download on the manufacturers or the vendors website so that  needs a little bit of porting help I see a couple of port is in the back maybe that will be  a thing to work on MongoDB there's a part available but it's not the latest version so currently  there's four of zero five and in ports this version three point six and of course the researchers  and the professors in my university are always interested in using the latest versions not  that they want to use it every semester the newest version because they usually also want to make  sure that their slides are staying valid for a couple of semesters so but in generally if you  are writing research papers you always want to test with the latest versions of the software  because otherwise they will not accept your paper or laugh at you the spark server itself we started  that last year and that's kind of involved the actual playbook that I wrote to deploy this has  actually over 180 steps and you can imagine how complicated it is because spark also includes  the hadoop installation and so there's multiple dependencies there to you know make it all work  okay so how do we start basically or what's the environment that I'm also working on so education  is an interesting place to work in in that there's no production environment or the production  is an environment is less important you would say than in a like a commercial environment where  you have to have 24 seven uptime and all these things so my production environment usually  only lasts like one semester which is like three to four months and then I basically can reinstall  again because that's all the software being used in the next semester so I only have to take care  of the software for like three months and then let's reinstall the whole thing again that I made  so much work on to make it running and so lectures as I told you they tend to stay with all the  software versions a little bit longer unless they're doing research in that case they need the  latest and greatest versions of the no SQL software and the production as again is only the  time frame when the students doing their labs which is typically five to six times and then next  the other week is the other group so we have typically two lab groups and the cluster makes  it actually easy to not be bound to a certain lab because they can use SSH to connect to these  machines from home or wherever they are and don't have to be at a certain lab in a certain hour and  can only do it for like how an hour and a half before the next class starts that makes the  cluster relatively flexible in its use and the popularity since we have that introduced it's  very popular because students see that they can run a lot of beefy benchmarks on that or make a  lot more CPUs spin than they have at home and so the problem was when we talk about installation  life cycles and updates that we also have to make sure that the operating system supports the no  SQL databases and so for the longest time when they when you were still running on the Linux side  of things they were running on the Ubuntu 14.4 version and we couldn't get to Ubuntu 16 or 18  today because for the longest time Hadoop had a problem with Ubuntu 16 you couldn't run a Hadoop  cluster on Ubuntu 16 that was the vendor trying to get it working and we were just waiting and  nothing happened and so the students kept asking hey can I upgrade this thing I want to use the  newer version if you want to and I said well if you want to use Hadoop there's no chance you can  try it and you would help me actually make it work but several students tried and never got there  and then someday Hadoop folks got it working I don't know what was the showstopper there but  then we could update to the latest version so on previously no problem latest versions perfectly  supported no problems but other software was kind of difficult in that regard because it was always  just available for Linux and BST is not even a download option on their website so that's a  general problem and I also give away these lab machines or these cluster machines to students  to do their thesis work on and it's always okay do you student what do you need as an operating  system well I use Ubuntu whatever at home and please install this on the cluster okay sometimes  I tried to install free BST for them and then ask them or gave them access to the note just to  get an email five minutes later oh I don't know this operating system can you install Linux for  me okay but a couple people tried and they actually did their thesis on the on those machines and  yeah even on free BST because what they're mostly running are applications they don't care what  operating systems are running on them and that way I can introduce free BST a little bit on the  sneaky side and some people just want to run their databases and don't care what kind of operating  system is running below okay so that's what the environment is that I'm working on or what I've  been trying to make changes in and so the changes have been coming slowly but steadily so the  rational here is I don't want to just introduce free BST because I like it I want to saw a couple  of things where free BST could actually improve things or make things better and that's why I  try to bring free BST in this environment and create something valuable for people so that I  don't maybe they don't see the features or I've never heard about the free BST features and that's  why I try to introduce this and told them about the cool features for example that ZFS provides  or that Ubuntu or that Hadoop is running quicker on ZFS than on Ubuntu with a traditional EXT  file system and so the thing that I told you about at the beginning that we were using the  actual lab image for the cluster which is more as a server environment that includes we get the  whole desktop craft from the labs into that environment so I had printer drivers on a server I  had graphics drivers sorry Nicholas on a server that will never run anything graphical because you  need the CPU power to do the actual benchmarking and so I took that image and actually was cutting  and slashing stuff out of it which I don't need and that actually reduced the install time of that  image from 25 minutes to 11.5 roughly and I think I can cut this down even further removing  more Linux stuff that we don't run on the cluster anyway and the free BST install itself just takes  three minutes probably less if I can tune it a little bit more because the only thing that I need  is basically get the OS running partitioning and ZFS setup and all that and set up the networking  and maybe some LDAP so that the students can log in with their student ID and that's pretty much it  then I reboot I don't have to install software in this part I can install software later using  Ansible or other software distribution or just package install that's pretty much it I don't need  this as a full big image that I have to copy over the networking to the to the node which  actually creates already some value because install times are going down the nodes are available  faster again if a student is finished and the next student is already waiting for that node to  become available I can say sure in 25 minutes or now in like 11.5 you can have the next node  available because that's the install time that it takes and if I'm testing things and I need to  reboot the machine because I totally messed it up and then I have quicker test cycles this way  so first steps if you're in a new environment and want to try out free BSD and maybe it's a hardware  or hardware environment it never worked on is the question of course is your favorite BSD working  on this it doesn't have to be free BSD could be other BSDs as well but I want to use free BSD and  so after the first boot I was kind of encouraged because a lot of devices were detected and  because free BSD is the actual server that's where free BSD is coming from it's a server desktop  or server desktop server operating system and there's a lot of drivers available for you know  these bigger and more expensive hardware parts and then the question is well I can install it but  that's the machine actually reboot into the new operating system or is it just forgetting about  bootloaders and all the other things that are required and luckily it did so then I knew that  this machine could run free BSD then what I did is was quickly start and a process to automate the  whole in OS installation because as much as you like your favorite operating system there's nothing  more boring than just installing it all over again it's clicking the same prompts entering the same  information all over again and so that's the first thing that I did trying to create this little  ISO image that's completely automated just load it and let the machine boot from that and it  automatically installs and you just have to watch and drink coffee and so also network setup is a  bit interesting in the university environment we do a little bit of IPv6 there we also have a  couple of things where we try to separate networks from the students so that they don't break into  I don't know our central servers and change their grades or whatever so these are the there's a  separate network for the big data clusters and it's also a bit secure this way and we have an  LDAP server as I mentioned that is providing the whole accounts for the department and for the  students so that the students don't have to create their own accounts and remember their  passwords all the time so it's just one password they have to remember and of course the NFS setup  from the central file server to mount the home directories for the students and that is imported  from from them and then they have every file that they have in the home directory on each note  that they have access to and of course I could reuse all of the many of the things from the Linux  desktop environment that they provided initially but I just had to move that to BSD where paths are  different or where some Linux specific things that don't work on BSD's and that's the actual  first steps that I took to create a note that is complete free BSD note and we're still not talking  about the application showing it this is just operating system and networking okay so then we  need to tackle bigger problems because then we have already established some kind of foothold on  the on the cluster with the BSD you know that it's running and then the question becomes is the actual  software that the professors are using running on that environment and again this I get a lot  the first sentence here well actually we don't care where so this is the database group where  the cluster is located so and they pretty much don't care what operating system they're running on  because they're mostly focusing on the database side and don't care what operating system is  running below and as long as their favorite noise grid databases are working then they can just do  their benchmarks and whatever they need for their research purposes and so the question is why are  you not just running everything in beehive because then you can have a bit more flexibility and just  providing Linux machines that are actually running on a free BSD environment and you just fake that  there is a Linux distribution in there and they are just running their same environment that they  are used to and that could be done the problem is that the professors actually or the students that  are working on that are actually doing performance benchmarking and creating papers and thesis work  and as much as I like beehive and its performance or its thinness in the layer between the operating  system and the virtualization there's still a little bit of overhead and that little overhead can be  seen in benchmarks or could be seen in benchmarks and that might skew the results a little bit so  it's kind of difficult to say okay let's fake everything in beehive overall we had used beehive a  couple of times because as I said the cluster was becoming very very popular and a lot of users  wanted to use the cluster and I said oh sorry I'm out of notes I don't have any notes left we have  to wait until the next student is finished with his thesis or her thesis and so what I tried to do  is give everyone a little bit of space on the cluster in a little beehive because they don't  usually run everything on all notes or only need like four CPUs instead of eight or only 16 gigs  of RAM instead of 32 so I could try to partition this a little bit and that worked a little bit when  we were in the situation where a lot of students wanted to do the data science labs and we didn't  have enough notes left so I had to create a couple of beehives to make that a little bit better to  allow more users because we want to let the users use our node or our cluster as much as possible  okay so that's the research part why we cannot do the beehive thing the other thing is you saw  that there is there are four disks in there and it's immediately clear that everyone says well  of course I created a rate Z rate 10 on that and perfect over four disks that's a good performance  boost you also would get ZFS compression and all the other things to manage your IO better and all  the cool things that Alan Jude could probably talk about and and of course you would also use a  separate data set for each node SQL database it's much more flexible you have much more ways to  manage your storage this way problem is same argument when they do scientific measurements or  writing papers apparently if there's a rate layer involved somehow then the papers that you are  submitting are flat out rejected the paper committees just look for and not just looking  for that but if there's a rate layer somewhere if it's a hardware rate or a software rate then  they just say well it's all all the benchmarking results are the result of the rate layer and  not of your tuning mechanisms in the database layer and that was kind of a bummer for me  because I really want to introduce ZFS into this environment because I think the professors would  also like that and but when they have to do this paperwork or this research paper things and are  rejected from conferences we didn't have that yet but that's the the downside of this if they  wouldn't do any benchmarks then it's perfect it's the environment we're probably going to use and  then I always thinking well if the paper committees reject those papers then you know how realistic  or production ready is this kind of setup because then you could say well if you cannot do a rate  then you just have to use a single disk but that single disk will fail I mean the disk in our  cluster are like five years old now so I just wait every hour to get a result from the smart CTL  to tell me that the disk has died but is this actually I mean this might be good for research  but is this production things that we're talking about here so I'm kind of a little bit on the  edge about that okay so next thing yeah the comment was here that the comparison is difficult if  you're different you have different rate setups and different rate levels for comparison okay so  now we have a free BSD a couple of free BSD machines in there next to the Linux machines that are  already running the old kind of hardware and the old kind of software stack so now let's grow and  extend a little bit so the biggest issue here is the file server oh we don't only have notes we  also have a file server and since the file server has been serving these home directories to everyone  and the research data set I mean we have a 370 gigabyte JSON file to load into the no SQL  databases to I mean it compresses super well on ZFS or on I mean G-SIP but they don't have to they  always have to ship the 300 gigabytes JSON file from one machine to the next to load this into the  no SQL databases and then do some kind of schema management or I have no idea what they're doing  but it's I mean 370 gigabytes of JSON okay and the file server is basically providing all these  data sets and yeah welcome to the new world so since the file server has been serving all these  files all the time there was no time to actually make it an upgrade of the software and to the  actual operating system bits and again the file server wasn't set up by me and what they did they  basically did a BIOS rate five so this is a hardware rate five and oh my god I wanted to run smart CTL  on this thing and it gave me nothing because it couldn't look into individual disks made up of that  rate five so I had no idea whether this disk were all dying or whether they are healthy so I did only  see this one big blob of storage that they were providing down from the BIOS to the operating  system and there was one issue from the start there were always so you would create a file on  your home directory or on the file server shared and it will always have strange permissions you  would get permissions like suddenly you had permissions that the dean was actually owner of  that file or that the secretary was the group of this directory and I was like where is this coming  from I mean the all the data is provided by the NFS or the L-LAP servers the L-LAP has all the  group information and the group sizes or the group IDs why are they so wrong when it just  touched a new file and so I after a while of looking at this I traced the problem down to the  etc at user conf on linux previously don't have doesn't have that file maybe that's why I didn't  find it right away and in there you have a value called first UID and if you're on an L-LAP setup  you need to increase this a little bit so that the local accounts are distinguished from the ones  provided from the L-LAP server and my predecessor didn't know about this or at least didn't do  that because the the timestamp on the etc at user dot com was from the date when the server was  installed no changes since then and once I bumped this all the files that were created had the  correct ownership and permissions but you can imagine this was like five years into the big  data cluster is live I'm not going into each individual's home directory and changing permissions  that way so I thought okay let's this is a failed experiment let's reinstall the whole file server  otherwise we're just coming into problems again and again I didn't like the hardware rate so I  always wanted to use ZFS for this so I made a move into reinstalling the file server oh here's  the first picture by the way whoo yeah so the process of migrating the file server was actually  quite easy so I said okay one of the nodes is now becoming the new file server because I couldn't  reinstall the file server in place even though we had relatively large semester breaks so where  this could be possible but even during the semester breaks there were students writing their  thesis on them or professors were doing projects on that so it always had to be something going  on in the file server so I said okay now I take a note from the cluster and I install it the way  I like it like ZFS and test all the things that I wanted to have that the later file server  were going to was going to become and I copy all the data files over and create the mount points  for the students and then let this run for a whole semester so this file server was actually  offline no one was accessing it and they were just making all the calls to one of those nodes  which was now the new file server and then during that time there were some issues or some things  where students were confused where the file suddenly ended up but all these things went into  basically a little file where I wrote down things that I needed to change or to make adjustments for  the actual file server installation and once the semester was over which was pretty much  painless so far I mean there were a couple of things that those were easy to fix and encouraged  by those things I did the actual change to actually now say okay a big announcement to the  whole department or the whole users from the big data cluster next week we installed the file  server and surprisingly no one else was complaining because they were actually using this one note  that I dedicated for as the new file server or the interim file server and they said yeah okay I'll  give you one day and if everyone is complaining then well we still have the old data on the  this one note that I used for it and so one day me very excited going into the server room getting  in front of the file server and removing the old Ubuntu installation and booting 3bsd for  the first time on it in which cause I removed the isn't it the nice the nicest thing to just go to  a server and remove the hardware rate without having to worry about the data on it so that was  cool so what I did is I created individual for each disk we have 24 of those in the file server so  for each individual disk I created a rate zero because you couldn't do a no op export into the  operating systems I had to so each disk has now is in a rate zero configuration exporting itself  to basically the operating system and ZFS will basically pick that up and create the pool out  of it and the data sets on top and ZFS installation went pretty well and in its cause I discovered  the amazing MFI util so that's basically managing all the things on the operating system side that  you would only do from the server BIOS and there are utilities like check each individual disks or  run periodic checks and that's a very powerful tool and of course then after I was happy with the  configuration in the file server setup that I wanted to have I copied all the data back from  the node that was currently serving all the files and then said well thank you dear department now  we have a ZFS based file server that I immediately saw the compression ratios and I was like yes so  much disk space was saved and yeah everyone was interested in that and I told them about of course  the other features that ZFS has quotas and reservation and all the other things because  that's actually the interesting part then the students don't overwrite their own home directories  anymore and I can limit the students to only say like two gigabytes of disk space that they're  allowed to use maybe maybe four if I have good day but yeah that's pretty much the actual migration  I'm pretty happy that I could make it work because the operating system on the file server is  really getting old and I was worried that someday someone might be hacking that thing so I'm not  actually too worried about external attackers I'm more worried about the students in computer  science trying to do oh what's this utility doing and oh now I have a cluster machine on my control  yeah not gonna looking good and then this is our new setup so you can see that I now export the  NFS I also did an upgrade to NFS for because now I can serve it from ZFS and that way I can see  what kind of data the students are actually saving in the home directory and also how much  disk space it takes and all ZFS automatically compresses all that so we have like home per  slash home is the home directories for the professors and the employees like me and they're  separate from the students home directories in case someone else has the same name as the  professors and they have their same home directory interesting concept and we have of  course home data sets that has all these crazy 300 gigabyte JSON files and other things that are  needed for the data science environment and yes no wrong permissions anymore because ZFS is  understanding everything that the LDAP file server or the LDAP server is telling them about  groups and permissions and I'm super happy now about this file server setup but there's more  there was no monitoring when I started this big data cluster administration or the app there was  previously a monitoring but it broke somehow and they didn't have the time to fix that so now we  have a since the file server is now monitoring each individual node and to make sure that the file  server who's running the monitoring isn't going down I have a little raspberry pi running  free BSD on ZFS that's checking whether the file server is still there you know watches the  watchers and so I get performance data from these nodes and if a student is making some crazy changes  like running a Bitcoin miner which has happened once that's probably my talk next year yeah then  I get the performance data and I see if a node goes down unexpectedly or if the network is having  problems so that's kind of nice I'm using Isinga for that it's not it's not Nagios but I like  Isinga the things that I saw and I will probably give a tutorial about Isinga in one of the future  BSD conferences and it's a nice thing because I get a little bit more email this time every  morning because each server is sending me information about you know how healthy the disks are and  what kind of processes were running and how many students have tried to log into this one of course  few futely and yeah this is the actual file server setup that we have and the monitoring that I added  to it and again it's all now we have a free BSD file server we have a couple of free BSD nodes  like 50 percent of the nodes are now free BSD and the actual monitoring is redundant with another  Raspberry Pi node that is just checking whether the file server is still online and so in summary  this is where we came from on the right side we had a desktop image based node installation  with coupon to 14.4 with reinstalled times of 25 minutes we had NFS free mounts with some crazy  ownership and permission problems we now had they had back then a little bit of Ansible scripting  like create a user on the file server or create a home directory oh not not so not so much but  it was at least a start and they had a broken monitoring back then and now with the free BSD  setup we have a fully scripted server-centric installation we have much lower installation  times this is just getting a coffee on free BSD and you want to take roughly 12 minutes I guess I  can shave a little bit more on that but I have to see whether the newer versions which are probably  now with more software that no one needs on a cluster we also have a ZFS based NFS file sharing  server which provides ACLs and all other things that are interesting in the NFS4 world I'm playing  a little bit with parallel NFS because that's been added to free BSD recently because that might be  giving us a bit of performance boost and maybe a couple of redundant files over in case the file  server might go down one day and I have pretty much everything Ansible scripted so all the Hadoop  installations are there the MongoDBs the couch bases that we're using on the nodes I just have to  say please install this on X machines and it gets installed and a couple of maintenance scripts like  delete all the users from last semester and you know get a new boot environment installed and  or snapshot done and things like that and most importantly we have a working and redundant  monitoring in the learning system that keeps me awake at night telling me oh this one machine  just went unexpectedly down and it's probably the disk that is five years old yeah so in retrospect  it was a little bit sneaky where I introduced free BSD so I didn't actually say hey tomorrow I  reinstalled the whole machine with free BSDs because that wouldn't work I would have gotten a lot of  pushback from the professors who were used to their working environment and didn't want to have to  suddenly deal with a new operating system that they've never heard of or I've never used and so  you have to be a little bit sneaky at times and also patient I mean this is a process that's been  going on for like the last three years and each semester I had a little bit more and more free  BSD machines in there and as long as students or users are only using applications most of them are  just oblivious they don't care what operating system they're running on there might be difference  environment different environments and this is a very unique environment as a university  but I was quite happy and this is not the end result the process is still going on as I had to  experiment with other noise-cured databases on free BSD and seeing what kind of versions are  available in ports. I collected a couple of tips and learnings at the end so as I said start small  and incremental do don't do the whole exploration step into different environments that you are  not familiar with start slow make incremental improvements and don't do a big bang thing of  saying okay next week we're all running on a new operating system surprise and also let people give  people the time to experiment with their operating system that they're now being given or their new  way that their home directories are provided for example and let them just get a little bit of  experience with that and once they have no complaints then you can just continue with the next step.  Always testing is a big thing test everything before you roll it out so if you're not sure  whether things work test again test again test different scenarios so for example for the NFS  mounts I not only had to test whether the mount works on free BSD but also working on the legacy  Ubuntu systems that we're using and there were some interesting things coming up so that also  needed to be tested and write everything down if you're a sysadmin you're probably used to that so  making notes about every config file change everything that you're just doing on the command line  paths that might be different from your other operating systems any settings that you might do  in your loader.con for rc.conf all these things might become important when you're doing a new  server installation and back everything up oh my god if I would have lost all these datasets that  are used for research I would have not my head again so if I lose some of the data that during  the process of migrating the file servers that would not go well with the professor so that  that way I backed up everything to multiple nodes to just be sure if one node would die I still had  the data from that but I would encourage you to experiment and as long as I mean we have boot  environments we have all kinds of mechanisms to save our necks and you never know what comes out  of that I mean if I wouldn't have installed the file server I would have never in discovered the  MFI utility and what kind of cool things it provides without going into the BIOS to make changes to  any kind of settings for the rate controllers and again don't just do things because you like  that operating system of yours make sure that you provide some kind of value to your users be that  they don't have to buy the server hard disk in the next couple of months because you saved a lot  of disk space using zfs compression or that they are now having the proper files and permissions  that they are used to small things like that can make a lot of people smile and your users are happy  that you are such a good sysadmin taking care of these systems sometimes you have to be willing  to compromise like the systems that I'm running is now doesn't have the zfs setup that I'm actually  initially permitted provided because that was the research and benchmarking things for the papers  that wasn't going to happen but it's the environment that I'm working with and I hope that if we don't  do any projects with benchmarking then we're not going to run this other setup and the rest is just  common sense really and my time is up and I guess you have a couple questions thank you  we have a couple of things about the cluster here if you're interested in a hardware and some of  the environment then find that link thank you
Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_0_155.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_156_1187.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_1188_1511.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_1512_3299.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_3300_3347.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_3348_3527.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_3528_3587.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_3588_3731.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_3732_3851.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_3852_3911.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_3912_7823.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_7824_9323.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_9324_12839.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_12840_18575.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_18576_18635.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_18636_18683.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_18684_18731.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_18732_22475.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_22476_26687.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_26688_26759.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_26760_26879.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_26880_27035.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_27036_30551.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_30552_33791.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_33792_34307.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_34308_34355.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_34356_34403.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_34404_34451.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_34452_34511.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_34512_34559.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_34560_34679.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_34680_35663.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_35664_35711.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_35712_36359.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_36360_37511.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_37512_38783.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_38784_39551.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_39552_41819.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_41820_45755.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_45756_45827.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_45828_46115.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_46116_47711.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_47712_50279.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_50280_50327.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_50328_51023.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_51024_54695.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_54696_59699.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_59700_59975.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_59976_60215.png Migrating_a_Big_Data_Cluster_from_Linux_to_FreeBSD_Slowly__but_steadily_T_nz63mnlZU_60216_60328.png
So, hello. Thanks for coming to my talk here in the BSD Dev Room. Some people might remember last year speaking, me speaking about my engagement in speaking or in teaching BSD in my university and this is the actual sysadmin part of my role there. So, it's a nice combination because I can teach what I learned as a sysadmin and vice versa, I can teach the students a little bit what's involved. So, today I'm talking about migrating a big data cluster from Linux to free BSD slowly but steadily. So, this is going on for quite a while now and I want to give kind of a status report and also a little bit how I approach the whole thing. This is what I'm trying to cover in the 40 minutes that I was given and don't worry it's a little bit more technical and also a couple of social aspects are in there. So, why this talk? Sometimes we get at the foundation or at the free BSD table downstairs the question of yeah, I like BSD, I want to use it not only at home but also in my workplace, wherever that is but I'm the only person using BSD there. I'm surrounded by other people who use other operating systems, what can I do? And so, I thought well this is actually what I'm experiencing or have been experiencing. So, I thought let's create a talk about this and this is an experience report basically how I approached it and maybe you can get some ideas how to implement some of this in your own environments because we want to see BSD rising in the not only in the private space but also in the environments where you work or in the work spaces. So, as I wrote here there's no one true approach so I have no master recipe to give you but I have my story to give you which is not also a successful one but also fraught a little bit with peril so it's not the perfect approach but at least it's one way to get slowly but steadily a little bit better. So, the first thing that you do is look around and learn because if you're starting in a new company or you're with a company for a while then you make a certain, you look around and see certain things that you might like or certain things that you don't like and maybe you figure out okay this is where BSD would come in handy in this area that is basically making a list of things that you want to change. So, I use it also to describe to you what the environment is that I work in and yeah so this is the first slide here so one day I inherited a big data cluster at work. So, I started there and a colleague of mine had actually started this big data cluster project that they started from the need that the students wanted to do long-term measurements of performance like for no-escalar database, benchmarking or thesis work or just projects that they were doing and so they were granted a couple of machines as called big data clusters it's basically a rack of machines I have a picture in the next slide and so they built it up I wasn't involved in that part so I basically had to take what they left me or what they gave to me and so I took over this this is the big data cluster it's called and it's using it's used in the University of Applied Sciences in Darmstadt Germany where I work in the computer science department although nowadays the whole university can use this cluster and the person that initially administered was a Linux person oh how new is that and was using Ubuntu at home so you thought it would be a good idea to use Ubuntu that he knew in this cluster as well and so this might be one of the first mistakes that were made but I'm getting to that and so this is an old Linux environment so actually the whole university is mostly Linux or the computer science department couple of people are now getting more into the BSD's now that I did a little bit of advocacy work but basically back then this is a couple years back there was no BSD at all maybe me running this as virtual boxes or teaching students to use BSD in my classes but that's pretty much it so the US image that they were using was based on the actual computer lab machine image which is kind of weird because this is a server environment and lab machines are typically sitting there waiting for students to do some work on them so lab machines are typically more desktop oriented and the image is also desktop oriented I don't know why they choose to use that on the big data cluster but I'm getting to that later on and then the classic thing operating system on the first disk and the next three disks that are in each of those nodes are used for the NoSQL database data files and they also bought a central file server for exporting the home directories for the students and the professors via NFS-3 so this is fairly straightforward and fairly easy to use so here's the little for the graphics or for the hardware geeks here out there so this is what we use so we have old nodes these are the ones that I started with so it this model the C6220s aren't even on sale by Dell anymore because they went out of support a couple of years ago and we're now using the next generation models the C6320 and so it's basically 64 gig RAM machines 8 CPUs and 4 hard disks that's pretty much it what we're working with and so some of the newer nodes have a bit more memory and a couple of newer modern processors but that's pretty much the whole the only difference and the file server is a little bit less in the memory area which is kind of bad if you're thinking about using ZFS later but pretty much the CPUs are there with enough power and they have enough disk space to cover the computing needs or the NoSQL database needs that we have so the software in use when I took it over was basically a dub installation and they were using also pig which is kind of accelerating the hadoop queries a bit and couch base was in use MongoDB and nowadays we're also having a spark installation which has a couple of other components in there so this is basically all in the area of big data and data analytics and we have a study program now with the mathematics department and that's called data science so the student can take that four year or four semester course and become data scientists and they're using the cluster basically for lab work thesis work or project in this area with companies for example and so then I looked at okay let's look around what actually is supported on free BSD so it's the dub support on free BSD answer is yes and I wrote about this in the free BSD journal a couple of months ago how I use free BSD and how you can deploy free BSD or hadoop on free BSD with ZFS because the ZFS part makes it actually interesting couch base itself has no native BSD client there's only lip couch base in ports and there's nothing to download on the manufacturers or the vendors website so that needs a little bit of porting help I see a couple of port is in the back maybe that will be a thing to work on MongoDB there's a part available but it's not the latest version so currently there's four of zero five and in ports this version three point six and of course the researchers and the professors in my university are always interested in using the latest versions not that they want to use it every semester the newest version because they usually also want to make sure that their slides are staying valid for a couple of semesters so but in generally if you are writing research papers you always want to test with the latest versions of the software because otherwise they will not accept your paper or laugh at you the spark server itself we started that last year and that's kind of involved the actual playbook that I wrote to deploy this has actually over 180 steps and you can imagine how complicated it is because spark also includes the hadoop installation and so there's multiple dependencies there to you know make it all work okay so how do we start basically or what's the environment that I'm also working on so education is an interesting place to work in in that there's no production environment or the production is an environment is less important you would say than in a like a commercial environment where you have to have 24 seven uptime and all these things so my production environment usually only lasts like one semester which is like three to four months and then I basically can reinstall again because that's all the software being used in the next semester so I only have to take care of the software for like three months and then let's reinstall the whole thing again that I made so much work on to make it running and so lectures as I told you they tend to stay with all the software versions a little bit longer unless they're doing research in that case they need the latest and greatest versions of the no SQL software and the production as again is only the time frame when the students doing their labs which is typically five to six times and then next the other week is the other group so we have typically two lab groups and the cluster makes it actually easy to not be bound to a certain lab because they can use SSH to connect to these machines from home or wherever they are and don't have to be at a certain lab in a certain hour and can only do it for like how an hour and a half before the next class starts that makes the cluster relatively flexible in its use and the popularity since we have that introduced it's very popular because students see that they can run a lot of beefy benchmarks on that or make a lot more CPUs spin than they have at home and so the problem was when we talk about installation life cycles and updates that we also have to make sure that the operating system supports the no SQL databases and so for the longest time when they when you were still running on the Linux side of things they were running on the Ubuntu 14.4 version and we couldn't get to Ubuntu 16 or 18 today because for the longest time Hadoop had a problem with Ubuntu 16 you couldn't run a Hadoop cluster on Ubuntu 16 that was the vendor trying to get it working and we were just waiting and nothing happened and so the students kept asking hey can I upgrade this thing I want to use the newer version if you want to and I said well if you want to use Hadoop there's no chance you can try it and you would help me actually make it work but several students tried and never got there and then someday Hadoop folks got it working I don't know what was the showstopper there but then we could update to the latest version so on previously no problem latest versions perfectly supported no problems but other software was kind of difficult in that regard because it was always just available for Linux and BST is not even a download option on their website so that's a general problem and I also give away these lab machines or these cluster machines to students to do their thesis work on and it's always okay do you student what do you need as an operating system well I use Ubuntu whatever at home and please install this on the cluster okay sometimes I tried to install free BST for them and then ask them or gave them access to the note just to get an email five minutes later oh I don't know this operating system can you install Linux for me okay but a couple people tried and they actually did their thesis on the on those machines and yeah even on free BST because what they're mostly running are applications they don't care what operating systems are running on them and that way I can introduce free BST a little bit on the sneaky side and some people just want to run their databases and don't care what kind of operating system is running below okay so that's what the environment is that I'm working on or what I've been trying to make changes in and so the changes have been coming slowly but steadily so the rational here is I don't want to just introduce free BST because I like it I want to saw a couple of things where free BST could actually improve things or make things better and that's why I try to bring free BST in this environment and create something valuable for people so that I don't maybe they don't see the features or I've never heard about the free BST features and that's why I try to introduce this and told them about the cool features for example that ZFS provides or that Ubuntu or that Hadoop is running quicker on ZFS than on Ubuntu with a traditional EXT file system and so the thing that I told you about at the beginning that we were using the actual lab image for the cluster which is more as a server environment that includes we get the whole desktop craft from the labs into that environment so I had printer drivers on a server I had graphics drivers sorry Nicholas on a server that will never run anything graphical because you need the CPU power to do the actual benchmarking and so I took that image and actually was cutting and slashing stuff out of it which I don't need and that actually reduced the install time of that image from 25 minutes to 11.5 roughly and I think I can cut this down even further removing more Linux stuff that we don't run on the cluster anyway and the free BST install itself just takes three minutes probably less if I can tune it a little bit more because the only thing that I need is basically get the OS running partitioning and ZFS setup and all that and set up the networking and maybe some LDAP so that the students can log in with their student ID and that's pretty much it then I reboot I don't have to install software in this part I can install software later using Ansible or other software distribution or just package install that's pretty much it I don't need this as a full big image that I have to copy over the networking to the to the node which actually creates already some value because install times are going down the nodes are available faster again if a student is finished and the next student is already waiting for that node to become available I can say sure in 25 minutes or now in like 11.5 you can have the next node available because that's the install time that it takes and if I'm testing things and I need to reboot the machine because I totally messed it up and then I have quicker test cycles this way so first steps if you're in a new environment and want to try out free BSD and maybe it's a hardware or hardware environment it never worked on is the question of course is your favorite BSD working on this it doesn't have to be free BSD could be other BSDs as well but I want to use free BSD and so after the first boot I was kind of encouraged because a lot of devices were detected and because free BSD is the actual server that's where free BSD is coming from it's a server desktop or server desktop server operating system and there's a lot of drivers available for you know these bigger and more expensive hardware parts and then the question is well I can install it but that's the machine actually reboot into the new operating system or is it just forgetting about bootloaders and all the other things that are required and luckily it did so then I knew that this machine could run free BSD then what I did is was quickly start and a process to automate the whole in OS installation because as much as you like your favorite operating system there's nothing more boring than just installing it all over again it's clicking the same prompts entering the same information all over again and so that's the first thing that I did trying to create this little ISO image that's completely automated just load it and let the machine boot from that and it automatically installs and you just have to watch and drink coffee and so also network setup is a bit interesting in the university environment we do a little bit of IPv6 there we also have a couple of things where we try to separate networks from the students so that they don't break into I don't know our central servers and change their grades or whatever so these are the there's a separate network for the big data clusters and it's also a bit secure this way and we have an LDAP server as I mentioned that is providing the whole accounts for the department and for the students so that the students don't have to create their own accounts and remember their passwords all the time so it's just one password they have to remember and of course the NFS setup from the central file server to mount the home directories for the students and that is imported from from them and then they have every file that they have in the home directory on each note that they have access to and of course I could reuse all of the many of the things from the Linux desktop environment that they provided initially but I just had to move that to BSD where paths are different or where some Linux specific things that don't work on BSD's and that's the actual first steps that I took to create a note that is complete free BSD note and we're still not talking about the application showing it this is just operating system and networking okay so then we need to tackle bigger problems because then we have already established some kind of foothold on the on the cluster with the BSD you know that it's running and then the question becomes is the actual software that the professors are using running on that environment and again this I get a lot the first sentence here well actually we don't care where so this is the database group where the cluster is located so and they pretty much don't care what operating system they're running on because they're mostly focusing on the database side and don't care what operating system is running below and as long as their favorite noise grid databases are working then they can just do their benchmarks and whatever they need for their research purposes and so the question is why are you not just running everything in beehive because then you can have a bit more flexibility and just providing Linux machines that are actually running on a free BSD environment and you just fake that there is a Linux distribution in there and they are just running their same environment that they are used to and that could be done the problem is that the professors actually or the students that are working on that are actually doing performance benchmarking and creating papers and thesis work and as much as I like beehive and its performance or its thinness in the layer between the operating system and the virtualization there's still a little bit of overhead and that little overhead can be seen in benchmarks or could be seen in benchmarks and that might skew the results a little bit so it's kind of difficult to say okay let's fake everything in beehive overall we had used beehive a couple of times because as I said the cluster was becoming very very popular and a lot of users wanted to use the cluster and I said oh sorry I'm out of notes I don't have any notes left we have to wait until the next student is finished with his thesis or her thesis and so what I tried to do is give everyone a little bit of space on the cluster in a little beehive because they don't usually run everything on all notes or only need like four CPUs instead of eight or only 16 gigs of RAM instead of 32 so I could try to partition this a little bit and that worked a little bit when we were in the situation where a lot of students wanted to do the data science labs and we didn't have enough notes left so I had to create a couple of beehives to make that a little bit better to allow more users because we want to let the users use our node or our cluster as much as possible okay so that's the research part why we cannot do the beehive thing the other thing is you saw that there is there are four disks in there and it's immediately clear that everyone says well of course I created a rate Z rate 10 on that and perfect over four disks that's a good performance boost you also would get ZFS compression and all the other things to manage your IO better and all the cool things that Alan Jude could probably talk about and and of course you would also use a separate data set for each node SQL database it's much more flexible you have much more ways to manage your storage this way problem is same argument when they do scientific measurements or writing papers apparently if there's a rate layer involved somehow then the papers that you are submitting are flat out rejected the paper committees just look for and not just looking for that but if there's a rate layer somewhere if it's a hardware rate or a software rate then they just say well it's all all the benchmarking results are the result of the rate layer and not of your tuning mechanisms in the database layer and that was kind of a bummer for me because I really want to introduce ZFS into this environment because I think the professors would also like that and but when they have to do this paperwork or this research paper things and are rejected from conferences we didn't have that yet but that's the the downside of this if they wouldn't do any benchmarks then it's perfect it's the environment we're probably going to use and then I always thinking well if the paper committees reject those papers then you know how realistic or production ready is this kind of setup because then you could say well if you cannot do a rate then you just have to use a single disk but that single disk will fail I mean the disk in our cluster are like five years old now so I just wait every hour to get a result from the smart CTL to tell me that the disk has died but is this actually I mean this might be good for research but is this production things that we're talking about here so I'm kind of a little bit on the edge about that okay so next thing yeah the comment was here that the comparison is difficult if you're different you have different rate setups and different rate levels for comparison okay so now we have a free BSD a couple of free BSD machines in there next to the Linux machines that are already running the old kind of hardware and the old kind of software stack so now let's grow and extend a little bit so the biggest issue here is the file server oh we don't only have notes we also have a file server and since the file server has been serving these home directories to everyone and the research data set I mean we have a 370 gigabyte JSON file to load into the no SQL databases to I mean it compresses super well on ZFS or on I mean G-SIP but they don't have to they always have to ship the 300 gigabytes JSON file from one machine to the next to load this into the no SQL databases and then do some kind of schema management or I have no idea what they're doing but it's I mean 370 gigabytes of JSON okay and the file server is basically providing all these data sets and yeah welcome to the new world so since the file server has been serving all these files all the time there was no time to actually make it an upgrade of the software and to the actual operating system bits and again the file server wasn't set up by me and what they did they basically did a BIOS rate five so this is a hardware rate five and oh my god I wanted to run smart CTL on this thing and it gave me nothing because it couldn't look into individual disks made up of that rate five so I had no idea whether this disk were all dying or whether they are healthy so I did only see this one big blob of storage that they were providing down from the BIOS to the operating system and there was one issue from the start there were always so you would create a file on your home directory or on the file server shared and it will always have strange permissions you would get permissions like suddenly you had permissions that the dean was actually owner of that file or that the secretary was the group of this directory and I was like where is this coming from I mean the all the data is provided by the NFS or the L-LAP servers the L-LAP has all the group information and the group sizes or the group IDs why are they so wrong when it just touched a new file and so I after a while of looking at this I traced the problem down to the etc at user conf on linux previously don't have doesn't have that file maybe that's why I didn't find it right away and in there you have a value called first UID and if you're on an L-LAP setup you need to increase this a little bit so that the local accounts are distinguished from the ones provided from the L-LAP server and my predecessor didn't know about this or at least didn't do that because the the timestamp on the etc at user dot com was from the date when the server was installed no changes since then and once I bumped this all the files that were created had the correct ownership and permissions but you can imagine this was like five years into the big data cluster is live I'm not going into each individual's home directory and changing permissions that way so I thought okay let's this is a failed experiment let's reinstall the whole file server otherwise we're just coming into problems again and again I didn't like the hardware rate so I always wanted to use ZFS for this so I made a move into reinstalling the file server oh here's the first picture by the way whoo yeah so the process of migrating the file server was actually quite easy so I said okay one of the nodes is now becoming the new file server because I couldn't reinstall the file server in place even though we had relatively large semester breaks so where this could be possible but even during the semester breaks there were students writing their thesis on them or professors were doing projects on that so it always had to be something going on in the file server so I said okay now I take a note from the cluster and I install it the way I like it like ZFS and test all the things that I wanted to have that the later file server were going to was going to become and I copy all the data files over and create the mount points for the students and then let this run for a whole semester so this file server was actually offline no one was accessing it and they were just making all the calls to one of those nodes which was now the new file server and then during that time there were some issues or some things where students were confused where the file suddenly ended up but all these things went into basically a little file where I wrote down things that I needed to change or to make adjustments for the actual file server installation and once the semester was over which was pretty much painless so far I mean there were a couple of things that those were easy to fix and encouraged by those things I did the actual change to actually now say okay a big announcement to the whole department or the whole users from the big data cluster next week we installed the file server and surprisingly no one else was complaining because they were actually using this one note that I dedicated for as the new file server or the interim file server and they said yeah okay I'll give you one day and if everyone is complaining then well we still have the old data on the this one note that I used for it and so one day me very excited going into the server room getting in front of the file server and removing the old Ubuntu installation and booting 3bsd for the first time on it in which cause I removed the isn't it the nice the nicest thing to just go to a server and remove the hardware rate without having to worry about the data on it so that was cool so what I did is I created individual for each disk we have 24 of those in the file server so for each individual disk I created a rate zero because you couldn't do a no op export into the operating systems I had to so each disk has now is in a rate zero configuration exporting itself to basically the operating system and ZFS will basically pick that up and create the pool out of it and the data sets on top and ZFS installation went pretty well and in its cause I discovered the amazing MFI util so that's basically managing all the things on the operating system side that you would only do from the server BIOS and there are utilities like check each individual disks or run periodic checks and that's a very powerful tool and of course then after I was happy with the configuration in the file server setup that I wanted to have I copied all the data back from the node that was currently serving all the files and then said well thank you dear department now we have a ZFS based file server that I immediately saw the compression ratios and I was like yes so much disk space was saved and yeah everyone was interested in that and I told them about of course the other features that ZFS has quotas and reservation and all the other things because that's actually the interesting part then the students don't overwrite their own home directories anymore and I can limit the students to only say like two gigabytes of disk space that they're allowed to use maybe maybe four if I have good day but yeah that's pretty much the actual migration I'm pretty happy that I could make it work because the operating system on the file server is really getting old and I was worried that someday someone might be hacking that thing so I'm not actually too worried about external attackers I'm more worried about the students in computer science trying to do oh what's this utility doing and oh now I have a cluster machine on my control yeah not gonna looking good and then this is our new setup so you can see that I now export the NFS I also did an upgrade to NFS for because now I can serve it from ZFS and that way I can see what kind of data the students are actually saving in the home directory and also how much disk space it takes and all ZFS automatically compresses all that so we have like home per slash home is the home directories for the professors and the employees like me and they're separate from the students home directories in case someone else has the same name as the professors and they have their same home directory interesting concept and we have of course home data sets that has all these crazy 300 gigabyte JSON files and other things that are needed for the data science environment and yes no wrong permissions anymore because ZFS is understanding everything that the LDAP file server or the LDAP server is telling them about groups and permissions and I'm super happy now about this file server setup but there's more there was no monitoring when I started this big data cluster administration or the app there was previously a monitoring but it broke somehow and they didn't have the time to fix that so now we have a since the file server is now monitoring each individual node and to make sure that the file server who's running the monitoring isn't going down I have a little raspberry pi running free BSD on ZFS that's checking whether the file server is still there you know watches the watchers and so I get performance data from these nodes and if a student is making some crazy changes like running a Bitcoin miner which has happened once that's probably my talk next year yeah then I get the performance data and I see if a node goes down unexpectedly or if the network is having problems so that's kind of nice I'm using Isinga for that it's not it's not Nagios but I like Isinga the things that I saw and I will probably give a tutorial about Isinga in one of the future BSD conferences and it's a nice thing because I get a little bit more email this time every morning because each server is sending me information about you know how healthy the disks are and what kind of processes were running and how many students have tried to log into this one of course few futely and yeah this is the actual file server setup that we have and the monitoring that I added to it and again it's all now we have a free BSD file server we have a couple of free BSD nodes like 50 percent of the nodes are now free BSD and the actual monitoring is redundant with another Raspberry Pi node that is just checking whether the file server is still online and so in summary this is where we came from on the right side we had a desktop image based node installation with coupon to 14.4 with reinstalled times of 25 minutes we had NFS free mounts with some crazy ownership and permission problems we now had they had back then a little bit of Ansible scripting like create a user on the file server or create a home directory oh not not so not so much but it was at least a start and they had a broken monitoring back then and now with the free BSD setup we have a fully scripted server-centric installation we have much lower installation times this is just getting a coffee on free BSD and you want to take roughly 12 minutes I guess I can shave a little bit more on that but I have to see whether the newer versions which are probably now with more software that no one needs on a cluster we also have a ZFS based NFS file sharing server which provides ACLs and all other things that are interesting in the NFS4 world I'm playing a little bit with parallel NFS because that's been added to free BSD recently because that might be giving us a bit of performance boost and maybe a couple of redundant files over in case the file server might go down one day and I have pretty much everything Ansible scripted so all the Hadoop installations are there the MongoDBs the couch bases that we're using on the nodes I just have to say please install this on X machines and it gets installed and a couple of maintenance scripts like delete all the users from last semester and you know get a new boot environment installed and or snapshot done and things like that and most importantly we have a working and redundant monitoring in the learning system that keeps me awake at night telling me oh this one machine just went unexpectedly down and it's probably the disk that is five years old yeah so in retrospect it was a little bit sneaky where I introduced free BSD so I didn't actually say hey tomorrow I reinstalled the whole machine with free BSDs because that wouldn't work I would have gotten a lot of pushback from the professors who were used to their working environment and didn't want to have to suddenly deal with a new operating system that they've never heard of or I've never used and so you have to be a little bit sneaky at times and also patient I mean this is a process that's been going on for like the last three years and each semester I had a little bit more and more free BSD machines in there and as long as students or users are only using applications most of them are just oblivious they don't care what operating system they're running on there might be difference environment different environments and this is a very unique environment as a university but I was quite happy and this is not the end result the process is still going on as I had to experiment with other noise-cured databases on free BSD and seeing what kind of versions are available in ports. I collected a couple of tips and learnings at the end so as I said start small and incremental do don't do the whole exploration step into different environments that you are not familiar with start slow make incremental improvements and don't do a big bang thing of saying okay next week we're all running on a new operating system surprise and also let people give people the time to experiment with their operating system that they're now being given or their new way that their home directories are provided for example and let them just get a little bit of experience with that and once they have no complaints then you can just continue with the next step. Always testing is a big thing test everything before you roll it out so if you're not sure whether things work test again test again test different scenarios so for example for the NFS mounts I not only had to test whether the mount works on free BSD but also working on the legacy Ubuntu systems that we're using and there were some interesting things coming up so that also needed to be tested and write everything down if you're a sysadmin you're probably used to that so making notes about every config file change everything that you're just doing on the command line paths that might be different from your other operating systems any settings that you might do in your loader.con for rc.conf all these things might become important when you're doing a new server installation and back everything up oh my god if I would have lost all these datasets that are used for research I would have not my head again so if I lose some of the data that during the process of migrating the file servers that would not go well with the professor so that that way I backed up everything to multiple nodes to just be sure if one node would die I still had the data from that but I would encourage you to experiment and as long as I mean we have boot environments we have all kinds of mechanisms to save our necks and you never know what comes out of that I mean if I wouldn't have installed the file server I would have never in discovered the MFI utility and what kind of cool things it provides without going into the BIOS to make changes to any kind of settings for the rate controllers and again don't just do things because you like that operating system of yours make sure that you provide some kind of value to your users be that they don't have to buy the server hard disk in the next couple of months because you saved a lot of disk space using zfs compression or that they are now having the proper files and permissions that they are used to small things like that can make a lot of people smile and your users are happy that you are such a good sysadmin taking care of these systems sometimes you have to be willing to compromise like the systems that I'm running is now doesn't have the zfs setup that I'm actually initially permitted provided because that was the research and benchmarking things for the papers that wasn't going to happen but it's the environment that I'm working with and I hope that if we don't do any projects with benchmarking then we're not going to run this other setup and the rest is just common sense really and my time is up and I guess you have a couple questions thank you we have a couple of things about the cluster here if you're interested in a hardware and some of the environment then find that link thank you
migrating a big data cluster from Linux to free approach the whole thing. This is what I'm trying data cluster project that they started from the to me and so I took over this this is the big in the University of Applied Sciences in Darmstadt Germany where I work in the computer science department couple of people are now support a couple of years ago and we're now using and they have enough disk space to cover the back maybe that will be a thing to work on and of course the researchers and the professors steps and you can imagine how complicated it is semester so I only have to take care of the do you need as an operating system well I use a process to automate the whole in OS providing the whole accounts for the department NFS setup from the central file server to mount the home directories for the students and that is file that they have in the home directory on each that they are used to and that could be done the papers and thesis work and as much as I like this a little bit and that worked a little bit compression and all the other things to manage Jude could probably talk about and and of course setups and different rate levels for comparison grow and extend a little bit so the biggest issue file server has been serving these home of the software and to the actual operating this directory and I was like where is this you need to increase this a little bit so that but you can imagine this was like five years into but even during the semester breaks there were professors were doing projects on that so it the old data on the this one note that I used for configuration in the file server setup that I quotas and reservation and all the other things of the future BSD conferences and it's a nice nodes are now free BSD and the actual monitoring provides ACLs and all other things that are in retrospect it was a little bit sneaky where I I've never used and so you have to be a little bit more and more free BSD machines in there and process of migrating the file servers that would the environment that I'm working with and I hope interested in a hardware and some of the