Replacing Traditional Backup Systems with ZFS
Hendryx-Parker, Calvin
Consistent backups are vital to any operation. Making sure this process is simple is key to making sure it is done at all. Tools like Amanda and Bacula are sophisticated and complex tools that do the task, but they require care and feeding to ensure they work consistently. Leveraging the built-in features of ZFS such as replication and snapshots can greatly simplify the backup process. Using portable management tools such as zfSnap and zxfer to manage backups across your infrastructure ensures that backups are done and can be even more granular than the typical daily/weekly backup schemes. We will look at using these tools from desktops to the server environment and how to have your backups easily distributed to multiple locations using FreeNAS’s scheduled tasks and replication. We will explore how to easily keep rolling snapshots and replicate them to backup servers. Easily give the ability to rollback on a file servers to a snapshot taken within minutes instantly, but also allow long term snapshots to be kept and replicated efficiently to backup volumes. We will also cover deployment techniques to ensure that systems have the correct packages installed and configured via configuration management.
 Welcome you all to my little talk about replacing traditional backup systems with ZFS. My name is Calvin Hendricks Parker and I work for a company in Indiana called Six Feet Up. We have been around since 1999, which is kind of crazy to say out loud because it means next year is our 20th year in business, which is crazy. But the whole time has been on previous team. I'm a long time previous user. I've been using it since 1996 and has kept our company safe, our data safe. And I want to talk about some of the journey that we have done over the years, you know, through various backup systems and how we have backed up our data. But basically since 1999, we've run a cage, an or a rack or host data for others someplace. So we have to keep that data safe. And we've been successfully restoring customer mistakes ever since then. Good job on gender diversity. Oh yeah, we are definitely an anomaly. We are a woman-owned business. This is the CEO right here. Also my wife, very wonderful person, I had to say that. But yeah, we've done a good job of trying to be a diverse company. So if you're looking for a job, we are hiring, looking for a developer right now. So I think the key to all of this is backups will be done when it is easy. I said by every system in the world. More importantly, like simple. I think easy should probably be taken out of my vocabulary. I know that my system is cringe whenever I use that word. So if you consider a bad word and really should make sure that backups are as simple to accomplish as possible because then they'll be done on a regular basis. So a little bit of backstory of where we have been with backups. We started from year one with backups like any good company should, even a small company like ours. We're only 16 people right now. When we first started, it was just actually my wife Gabrielle in 1999. And so I started with Amanda as a backup system from 99 to 2012. We did a little bit of Bacula in 2007 as a proof of concept. And then dofolian on Bacula in 2012. And I'll talk a little bit about each of those systems. Has anyone in the room been using Amanda? Yeah, this is like. Yeah, Amanda is, I guess, the old standard of backup systems and open source. If you've not used it, it is pretty awesome because it is very easy to use. Some of the things I really loved about it is written in C in Pearl, has a BSD style and GPL license. It is currently backed and supported by a commercial entity. So there is people actively working on it. I would recommend it to people because it is still a good backup system. It was my first backup system I ever used for a different job before I started our company. They had Amanda in-house. It was a visual effects company. So they had tons and tons of data to back up for all the various movies that they were working on. So it was really important to them. And one of the things I really liked about it is due to native tool usage in Amanda, we were able to do things like recover from a very painful experience from one of our system and accidentally deleted the index directory of Amanda. So Amanda has an index that maintains. But if you lose that, you are not actually dead in the water. You just use tar and dump and you are on your way. I mean, it was painful, but we didn't lose any. There was no data loss, which was nice. So Amanda did a good job of protecting our data through the years, which is why we held on to it and used it for so long. I think, you know, as with any first love, there is issues that kind of caused us to go looking, you know, our wandering eye maybe. We stopped using Amanda due to the age of our current packages we had installed. So we were on freeBisD and things kind of kept moving. The upgrade from Amanda 2 to Amanda 3 ran into, it was kind of rough. There was lots of renaming, UIDs and permissions and there was various time-lots and random hangs. So we had a real hard migration trying to go from 2 to 3. So basically at that point we decided we needed to look for something that would be more stable and could grow with our organization. Another issue was that at the time, Amanda didn't support ZFS very well. And so we were very excited about using ZFS right from the beginning. I think since FreeBisD, it was available on 8 and we started using it like right away, even before it was kind of the standard thing you would even use as a file system on FreeBisD. And because of that, Amanda didn't work well with it to begin with. And Amanda also would commonly, one of the problems we would run into is it would commonly bring down some machines if you had rather large backups. So if you don't know much about Amanda, but it'll spool up a lot of data into temporary directories. So if it was short on disk space, you could guarantee bringing a machine down with Amanda trying to back it up, which is unfortunate. And feel free to ask questions along the way about any of our experiences with these tools. The next one I actually looked at was Bacula. Well, actually we did use. We kind of proof of concepted it and therefore about a year. And then when we finally got all those troubles with Amanda built up, we basically said we had to make a change and move into Bacula. Actually Dan was probably one of the reasons I went to Bacula because of all those great posts on the FreeBisD diary about using Bacula. So I had a great guide to get us into a good spot with Bacula. The back of this file system level backup, it used to be a GPLv2 and I don't think things improved by moving to the AGPLv3 or you can get a proprietary license. And I'll go into that a little more later. I think this should have been telling right off the bat. When you come to the Bacula website, they don't necessarily encourage you to use Bacula right away. This is right on the front page. I think Bacula is not a simple system. As you kind of saw in this, this is actually like an architecture diagram of how Bacula works on this screen right here. And there are lots and lots and lots of moving parts to Bacula. So if you are not used to lots and lots of moving parts or if you just have a very simple set of needs, Bacula is definitely not the choice for you. Now we were, we did have an infrastructure of a good enough size I think it deemed worthy of Bacula and actually Bacula's service very, very well. Again, restoring customer mistakes since 1999 and Bacula has helped us recover from many of them. But there's not, not without, Bacula has not been without some drama. Maybe not the drama like we saw earlier, but there's definitely been some friction with the Bacula community and the Bacula creator and what he called the hidden cost of creating open source. So Bacula moved to an open core model. There's an enterprise version of it which gives you the proprietary license. And one of the promises that they made was to any feature they include in Bacula Enterprise within some amount of time, a maximum of five year delay, would land then in the Bacula open source product. That hasn't always happened. For example, one of the main things we would like to have had in Bacula would be block level backups which is included in their open source version 5.1 which was released in 2011. And it's still not actually in the open source version as of today. So that, it just, this is kind of a bad example of open core gone wrong. And so I'm not a big fan of that. I definitely want to have more of an open source flexible solution where I feel like it's kind of a partnership between me and the community who's, you know, producing the open source software. So beyond the political issues of Bacula, and Bacula is a very good product, but I think maybe we had some issues with the, our use cases for how we backed up our data and the kind of workloads we were backing up specifically. One of the problems that we've run into is Bacula is, because of the various moving parts, there's lots of configurations. I'll show you some examples of how we improved upon that. But there's many, many configurations that at some point our configs got so bad and so un-maintained that we were basically down to having a once a week backup of all of our stuff because it was, things had just gotten out of whack. Bacula does require a database, so there's many moving parts. So the database is there to index the files that all get backed up, which is nice when going to restore because it's very fast to like get listings and find the files you need to restore. And in that case, you know, Bacula is very, very nice. But it takes lots of care and feeding to make sure that things are always running very smoothly. Now, in our case, we run various web applications and the database that backed those web applications, if they, they never use Zope in the room. I got a few people. Okay, so the default storage for Zope is a large file, system-based file, and that file can get very big when you go to file, file level backups. That whole file changes every day. Actually, it's changing every time the transaction happens in the database. So for us, every night's backups of any incrementals would always include the largest file in the whole system, which is typically that big database file, which is paying the butt. It means for very inefficient use of our disk space for the backups and very slow for reading and writing that, that giant file, maybe only a couple hundred K of that file may have even changed. But then we're backing up a whole many, many, maybe tens, if not hundreds of gigabyte single file. That's not to say that our application was perfect and could have been improved fun, but because of our specific use cases, it was very, very slow. Yeah, so with the slow to backup part, it does leverage a file daemon on every box. So if you're running back-in, you will install a file daemon on every client machine that you'd want to back up. And then there will be a director piece that actually tells a backup storage daemon to stream the data from the file daemon on the client back to the backup server or wherever you're storing, wherever you're either to tape or to some kind of disk storage. And to actually help configure our bacula, we ended up creating sets of Python code to produce templates and actually would run a set of scripts to generate our many thousands of lines of bacula configuration each time a change had to be made. So I think that there has to be a better way. Oh, funny enough, there might be a better way. What if, so this is like when you start thinking outside the box and actually looking at what you really need for a backup solution, you have to kind of analyze the various options that are out there and find one that fits your specific needs best. And for us, because of this web application that we typically develop or deploy to, how having a block level backup is pretty critical to us being able to do efficient and fast backups and making sure that they always happen. But what if there was a file system that supported some of these advanced features we were looking for? Now, for example, snapshots, replication, compression, kind of built in data integrity. I think something like this exists, which is exactly what we were looking for. And also, what if there was a way for all of our VMs and all of our jails to be backed up by a file system that supported that block level changes and incremental snapshotting and replication? All right, and did that at a block level. Did I mention block level? That's really important to us. Given that we were a FreeBSD and we love ZFS, this is a natural fit. So we have been using FreeBSD and it was nice is that ZFS actually is cross-platform. So depending on whether we had maybe a Solaris, or actually at some point we did have Smart OS, Open Solaris, FreeBSD. I don't think we ever had Linux running with ZFS, but that's actually quite stable now from some of the work that's been done. But it's nice because then it works cross-platform. We can actually use the same solution across all those various OSes. And actually we were able to greatly simplify how we were doing our backups. For the goals, and these kind of just apply anytime you're setting up and trying to figure out how to accomplish your backup tasks, is it has to be reliable. We have to make sure that we can ensure that it runs and if there's errors in the process that we get, obviously a report or someone gets alerted to go fix the issues that are happening. It needs to be granular. We need to be able to get down to a specific VM or a specific file directory and be able to restore a file quickly from that backup, especially if it's that large database file. Now we really care about integrity. How many people who have been on a Mac for any amount of time have run into the wonderful HFS plus silent file bit rot? Yeah. This has bitten me. I'm actually running, you know, prior to the high Sierra, ZFS on my laptop, exactly for those reasons. So I use Mac, or open ZFS on Mac. I've been very happy with it. And the integrity issues are awesome. So I kind of use it on my Mac. We also use it in all of our infrastructure. Another key piece here is having few dependencies. I want as little as possible to have to touch the VMs or the jails. They know as little about the backup process than they need to. And it needs to be easy to configure. And then obviously at the very end it runs with little care and feeding. So when moving over to using ZFS for our backup solution, the first and initial win we got was obviously going to be ease of use and simplicity of configuration. So this is on our backula setup. These are the amount of configuration lines that are required to backup. I think we've got, probably, like 300 hosts or 200 hosts inside this. And you can see here we're well over 1,000 lines of configuration to manage that backup. And if you weren't using some kind of automated scripting to produce this, this is very tedious and very possible, very error prone, and which led into our issues with backula being the config getting out of date and us basically getting down to like one backup a week. So every server that has to be backed up will have to have the backula fd process. So every server has this 27 line configuration file. And then the director has the others. You can see there the regular, like scheduler and jobs files. So we had to have, again, some way of managing this process and we had written a bunch of custom Python to be able to do this for us. But again, that falls out of repair, disrepair quickly. When moving into using ZFS for our new backup solution, instead of having every specific, every VM and every jail running the backula file daemon, we changed our, shifted how we did that. Instead of doing it at the client level, every client backing up, we then took every jail host and every beehive host and actually backed them up at the hypervisor or jail host level. Which again greatly simplifies things because you may have 30, 40 jails running on one machine. You may have 10 to 15 virtual machines on one machine. Now we're down to configuring one set of configurations instead of like 15 or 30 per machine, which is nice. So this is on the jail host, for example. We use a combination of ZFS SNAP and ZXFOR to do the snapshot management and then the replication. Now what's nice about ZFS SNAP and those tools specifically is that they are, they have very few dependencies. Most of them are just shell scripts that work cross-platform. The ZFS SNAP works on my Mac, just as well as it works on previous D, which is nice. But to enable the snapshot management, you just have to add a couple of lines to your periodic.conf and from there you basically enable it, tell it which recursive file systems you want to grab and then it manages all from there. This configures daily and weekly. You can actually go down to other granularities, but for us that was enough to satisfy our promise to our customers with how much we would retain for them, for their data usage. And so that's on the, so every, every jail, every B-hype host has this in it now. And then on every trusted server, so like for example the hyper-riser servers are considered trusted because we don't have clients logging into them, we would use ZXFOR to send back the incremental snapshots. So in CronTab there's one line and that handles sending all the snapshots. So ZSnap, ZFS snapshots creates the snapshots, ZXFOR on a once a day process will send them all back to our Freeness servers. So they're all running jail or we're all running the Freeness for our storage architecture on the back end and I'll show you some cool stuff we've done with the Freeness stuff later. But what's nice here, this is pretty easy to configure. Alan is the kind of somewhat maintainer of ZXFOR but it doesn't require a lot. It's very simple, very straightforward. You know, it can delete snapshots on the destination, it can force rollbacks of file systems in case it wasn't kind of ready to receive the snapshots for some reason. What else is here, it's interesting. And then the recursive data sets. So yeah, one line, we're sending them back everything to the servers. Now any server where we didn't necessarily trust the host, like maybe we did have some bare metal virtual or bare metal hardware servers hosted for customers, we don't want to put our keys on that host that could access our storage server. So instead of we kind of flip the direction, we would pull from that bare metal host back to the Freeness server instead of pushing from the servers back into Freeness. So in this case, it's just a simple SSH send receive SSH into the remote host and then send the receive back into the Freeness server. And that was just mostly because we didn't want to have our potentially private keys on some remote, who knows what server. So this is on the Freeness side of things, we basically have a cron job that will run that, this script right here. So we just basically use the scheduling, you know, right through the GUI so any admin can very easily set up and add in new machines if they need to or manage the very scheduled jobs that are inside of Freeness. Now what's nice is you get a bonus if you're using Freeness. She's running Freeness in production right now. Okay, so you guys all should be well aware of this. The Freeness has, I mean, we have multiple data center locations and it's very easy to do offsite backups. So if you guys haven't done this, you definitely should check out the replication tasks area of Freeness. Basically, it's super easy to now set up, remoting that data offsite to another Freeness machine and running in some other data center. All with, you know, one line through the GUI, you basically set up a schedule and point it at a data set and it can now send those things offsite. So it's kind of an added benefit to certain customers who want to kind of upsell into offsite backups of their data. It's very easy for us to just now come in and add one line per customer who wants to have that offsite feature enabled. Then very easy for the admins to manage and maintain. So this presentation is online in my GitHub, my email and my Twitter. So if anybody's had any questions, I am happy to answer them. No questions? All right, Dan. Instead of time-dropped, have you considered, let's say, periodic? Yes. So we do for part of it using the ZFS snap stuff, but I'm not actually like, you know, formalize that more. The reason I like periodic is it can be dispelled by a fork or something, and you're not messing around with a cron-tap, but it's just a disk. I know the cron-tap is a fist bump. And you can also enable and disable it with nrc.com. True. Yeah, so I agree. That's definitely a better way to go to add new periodics for the scheduling. If I can cron.d? Yeah. So we use, and so all of our servers are configuration managed by salt, and so we actually just have that as part of the default, you know, salt profile when a machine's brought up, this stuff just gets put in place automatically. Like, luckily there's no humans involved typically in getting any of this running. So new machines are automatically snapshotting and sending data back in day one. Could you give us an idea of the improvement in both the time performance and backups on the storage? Oh, yeah. I probably didn't mention that in my notes here. The back gala backups were getting to a point where they were taking longer than 24 hours sometimes. So we would have trouble basically doing them daily. That's what led us to the kind of bad situation. We did get it down to, say, like, 14 or 15 hours before we switch over to ZFS. And when the switch over to ZFS, it now back up the same set of servers. Once you've got the initial base backups and all the base snapshots sent over to the storage server, the new runs only take a couple hours, like maybe an hour or two, to run through all of our infrastructure and send the incremental snapshots. So it's tremendously faster and saves a lot more space on data. A little bit of a question. All right. How much, can you say, a couple hours to transfer to do the replication? How much bandwidth do you have between the data centers? Oh, for us, between the data centers, it's 100 megaline. But not every data set gets sent across to the remote data center. We only offsite for customers who want that extra feature and then also just for our stuff. Go ahead. When you were, when you were generally unhappy with various, one of the reasons that you were unhappy with was the commercial to community relationship. Did you guys look at various at all? We did not. I'm not actually not even familiar with that. Is that the commercial? Okay. Because of this. I mean, again, for us, the simplicity, it was a bigger win than almost the politics and the commercial licensing pieces of it. Luckily, we don't have to restore very often. But the restores are easy. You go into either the machine and send receive something back or typically it's one file. Somebody's like, I messed up a single file. And so we can actually from the free now machine just log in CD into the specific snapshot directory and grab the file, you know, write a file system. So in other words, the snapshots you send, you basically fully replicate file system on here? Correct. Now, we have to get the clone and mount that clone if you want to like go in and get the data. But yeah, we do. It's all of us just all I'll put back on the freedom system. What do you do about picking up old snapshots? So the old snapshots get cleaned up automatically from the zx for command. It will delete. Well, as the ZFS snap takes the snapshots, it cleans them up on the client's side. And then on the server side, the zx for deletes them if they don't exist on the destination. So from the client to the free now box. So one of the keeping library of old ones in case, you know, a file gets deleted and that machine gets copied over the snapshot. Right. So we keep we were keeping a month for a year and then weeklies for I can't remember what the time frame was. That's all configurable be the ZFS snap. Yeah Yeah Right Yeah Yeah Right Right Yeah, probably my biggest issue with, you know, snapshot management is the fact that ZFS is too good and it's hard to delete data if I needed to because of, like, if there's some kind of legality and someone required me to delete some kind of backup, I'm going to have a hard time actually with that. Yeah. Well, and I liked your suggestion during the BOP or the ZFS talk today was to encrypt those data sets, have encrypted data sets and so basically the deletion process is throw away that key. And I think that's a good idea for now. So one of my big fears with replicating snapshots where you propagate the deletion from the source, there's been cases where I accidentally deleted the snapshots on the source and I wouldn't want those populated to the backup. If they disappear, I'm probably going to have to fix quite some of that as a grandfather in the game there, but I would rather have a minimum sort of time before things go away as well. Yeah, actually, I would like to keep, I would like to guarantee keeping at least the last 24 hours with in case a certain idiot. That's a good idea. It should be pretty easy to add. Yeah. Although the next talk is... I was going to say the next talk, I think, is really, made me really relevant. It works very well, I must admit. I need to write it. I just need to make sure. Yeah. Don't be too too... I'm on the domain for a rewrite. You never started. No, the next... Actually, I talked to the next speaker about his Z-repl and it seems to be a much more formalized system around the snapshot and replication management piece, which I wasn't... It's brand new and he's kind of the only one using it right now, but I was very interested in actually picking that up and starting to use it as well because I think it may complement our set of tools really well. And now, the only thing I don't like about it is it adds in more complexity. This is already very, very simple and does accomplish the job, but if I can improve on it by... And the value is there in that improvement, then it's worth doing. Yeah, question. Yeah, so the question is how do we go about protecting against, say, a rogue system in or some kind of human error with an offline copy of the data? Right now, we just have the off-site, but it's not offline. That could very easily propagate some fat fingering across to the disaster recovery center. Yeah, there's the two cases. Something blows up that's fine, but malicious or accidents is a real concern as well. Yeah, I don't currently have an offline solution for that right now. I mean, one of the things we do... Yeah. I mean, for us, it is more important that we have the off-site had more up-to-date data though, too. Yeah. Yeah. Well, that's something we've thought about at the off-site center having a second Frenez box there that they do... The replica drops the data down, like, say, on a very specific schedule, so you have that protection of an additional copy there. Have you or anyone here looked at, say, two-factor authentication or even secure levels to prevent the pollution of snapshots? I haven't. And that's actually one of the things that I... Like, one of them... I don't know, not a problem, but like, one of my issues with ZFS is it's very easy and very quick to destroy. There's not even an ask you, like, if you care to destroy it. It's just... You've just destroyed it and it's very efficient. Thank you, Matt. But... I should almost... At least that on every machine, then. I could say for many times by the back, I almost always typo destroy. No one ever alias ZFSD story to destroy. But yeah, it is true. It's very easy to destroy things on ZFS. I have not looked, but once you destroy it, there's not like a, I can get it back type thing. It's gone. The snapshots, yeah, it's true. Which I have lots of snapshots, which is very nice. Yeah. So I'm going to say, how are you prepared to deal with, say, if a core telling you that a certain set of data must be destroyed from... Well, I'm saying it. I'm not at this point. It's very, very difficult to get rid of data in this setup. Luckily, you know, cross fingers. It literally happens better. Right? Yeah. So I don't even know, like, how we could go about that. You need to have a separate data set for every unit of data. Like, it took us a while to realize, but one data set for every single customer. Which is what we have. We do have one data set for every individual customer, because every customer has either a VM or a jail, and all the jails are, each of them have their own data sets. So a specific customer came to us and said, we need to have our data deleted, and yes, they can destroy all those data related to just their data. If they have a subset of their data, that's the only option to make them do VM. How can you get the data out? Right. That's probably exactly what I'd have to do, because it's just the data stays around. That, it has to protect you. You don't want it to. Yeah. Then you can put data sets. Yeah. I mean, the data sets are cheap, so use them, you know, all over the places, which is nice. Yeah. So, the way that I clean out unwanted data from backup snapshots is when I make a writable clone of the snapshot, then I eliminate what I don't want in there. You have to do that for every snapshot where the data is, and then you delete the original snapshot, and then rename, resulting snapshot back to the original name, then you actually have the right snapshot with the correct names. Yeah. So, you're saying it's possible to make a writable clone, delete the file, but you have to do this for every snapshot you would have. Could be, you know, we potentially have hundreds or thousands of snapshots. I would say that sounds like it's calling out screaming for automation. Yeah. That's actually one of my favorite features of ZFS is the.CFS directory. Yes. So, I don't know if you'll have an answer for this, but there's been very good news in the town. I've got some data where I have either contractual or regulatory requirements to put it on offline media, whether it be Word media or tape. Yes, I still do the tape world. How do you get ZFS data onto the tape efficiently nowadays? So, the ZFS data is a file, it's a serialized property. You can put it on tape and then receive it a year later. So, release ZFS and redirect to.CFS. Oh, pipe it to tar. Yeah. So, the question is how do you get ZFS data sets down to tape, and you can just stream it down to tape, ascend. Do you do the full under tape and you put it on the end of the issue? That's too easy. How would you like to set it apart? You know what tar stands for, right? Yes, that's true. ZFS has made our lives too easy. It's Michael. Are you using specifically LZ4, sorry, LZ4, and have you ever attempted deduplication that hasn't suited you? I've never attempted deduplication. We've had enough storage. That's not an issue. We are using LZ4. Nothing unusual? Nothing, I've never noticed. No. Is there something I should be aware of with LZ4? Yeah, there's something you're coming about. Well, I'm looking forward to the new stuff Alan is talking about. Yeah, that I'm actually quite excited about. So I'm happy to switch when the time comes. Yes. Have you given any issues? It sounds like a Zofi since most of the flat file data is. Have you given any cases where there's a process that's caching things heavily into memory and doesn't flush as often as it should? Where I mean to talk something like that. So the question is if we run into an issue where something is maybe not flushing down the disk as aggressively as we'd like with the application. It's not actually an issue with Zope. I mean now we're getting more into a Zope discussion specifically. Zope is an append-only database and it on start will automatically trim any corrupted transactions from the end of the database as it starts. So you can very reliably snapshot that file at any point in time. Even if the power outage happened or whatever it's very, very resilient to starting back up and basically rolling back whatever failed transactions might have been on the end of it. So for that we've just benefited because that's the nature of that application. If you have other databases where you want to ensure consistency like Postgres or MySQL I mean you should definitely have probably an additional cron jobs and in those cron jobs maybe to do some kind of a start backup, stop backup type thing. One more question and this is probably for Alan. Is there a way we can do this backup to force or if you have a ZIL to force it to flush? That I don't know. I think it's definitely a question for the right in front of you. What you're saying is the snapshot will be the same as the created one. Okay so the question is about the ZIL flushing on, but it happens automatically when you do a snapshot. And since you only send snapshots you have the whole data set. Always be consistent. I love that. Just to combine a couple of recent questions, slightly off topic data, but there's like an extra strategy for when they have backups on offline media like pay for CDs. And then they get a CDPR request telling them they need to remove all data related to the typical customer. I've only had a request like that once where we had customer data we shouldn't have had and it was in subversion. And so I had to filter it out of every subversion transaction to undo it, but I didn't have it on tape. We probably did, but we're using backill at the time. Yeah. Backill would have attended to that for you. So basically some of the data that I have to put onto archival media is government data. We have an unofficial opinion that as long as it is right only media, so word media, whether LTO or MDS, or anything like that, as long as it is not ultra bulk, the answer legally can be no, we can't do that. If you are able to offer the media, you are allowed to say no, we can't do that. So you choose your media carefully. This is yet to be tested. This is an unofficial opinion that we've been told verbally, but that's something you might explore through your legal team. Because almost all the legislation around the world allowing this does have some exemptions for where it is not technically possible to delete the data. And G8 is on optical media or something like that. Yeah. Couldn't you get into like a semantics thing if you have a check some stream? Absolutely. That's not changed. That helps. That'll be next year's presentation. You talk to your legal team, etc. But yeah, there are ways to do this that don't put yourself in the foot. And that's like the logical stream of like one database for each person. One database for every individual in Europe. Yeah. I don't know if your opinion law situation is, but this is under Canadian law and that's what we've been told. Whether it's mature or not, I am not a lawyer. Who in the room has gotten a notice to delete data? Okay. Coming. Well, not under GDPR, but under pre-project. Oh, actually, I'm sorry. We have season this, had season this since the years when the BBC project, the man with the most diverse workflows and stuff that people really want to experience. And do you have to go, you've gone all the way back through the backups of that? In that case, at the time, it was what backups, but... It was in CVSAR. Yeah. Globally distributed backups at the time. That's the official story. I'm sorry, the official line. So they do happen and yeah. I expected to get formal common works to figure out how GDPR is intended to be used. Yeah, this, I mean, I'm kind of worried about this because we do write applications that are portals where people log in and have a profile. And I don't know how to reliably say I've gotten rid of all your data. Like, there'll be these remnants. Or I can say, I don't know, with GDPR, do you have the time frame with which you have to guarantee it's all gone? I think the general rule is if you can make a case that what you're doing is very reasonable, they're more interested in the fact that you can do it and you're doing the right thing than specific doting guys and possibilities and specific timelines. Yeah. So you have a clause about, has to be commercially reasonable and you can't bankrupt yourself to leave the data. It's complicated not a lawyer. There's a lot of good write-ups on it. We do have a little case in talk to a lawyer. Discussion. Talk to a lawyer. Yeah. So talk to a lawyer is our answer. Almost makes you wonder if they didn't make this happen. Any other questions? Thank you all very much.
 Welcome you all to my little talk about replacing traditional backup systems with ZFS.  My name is Calvin Hendricks Parker and I work for a company in Indiana called Six Feet Up.  We have been around since 1999, which is kind of crazy to say out loud because it means  next year is our 20th year in business, which is crazy.  But the whole time has been on previous team.  I'm a long time previous user.  I've been using it since 1996 and has kept our company safe, our data safe.  And I want to talk about some of the journey that we have done over the years, you know,  through various backup systems and how we have backed up our data.  But basically since 1999, we've run a cage, an or a rack or host data for others someplace.  So we have to keep that data safe.  And we've been successfully restoring customer mistakes ever since then.  Good job on gender diversity.  Oh yeah, we are definitely an anomaly.  We are a woman-owned business.  This is the CEO right here.  Also my wife, very wonderful person, I had to say that.  But yeah, we've done a good job of trying to be a diverse company.  So if you're looking for a job, we are hiring, looking for a developer right now.  So I think the key to all of this is backups will be done when it is easy.  I said by every system in the world.  More importantly, like simple.  I think easy should probably be taken out of my vocabulary.  I know that my system is cringe whenever I use that word.  So if you consider a bad word and really should make sure that backups are as simple to accomplish  as possible because then they'll be done on a regular basis.  So a little bit of backstory of where we have been with backups.  We started from year one with backups like any good company should, even a small company  like ours.  We're only 16 people right now.  When we first started, it was just actually my wife Gabrielle in 1999.  And so I started with Amanda as a backup system from 99 to 2012.  We did a little bit of Bacula in 2007 as a proof of concept.  And then dofolian on Bacula in 2012.  And I'll talk a little bit about each of those systems.  Has anyone in the room been using Amanda?  Yeah, this is like.  Yeah, Amanda is, I guess, the old standard of backup systems and open source.  If you've not used it, it is pretty awesome because it is very easy to use.  Some of the things I really loved about it is written in C in Pearl, has a BSD style  and GPL license.  It is currently backed and supported by a commercial entity.  So there is people actively working on it.  I would recommend it to people because it is still a good backup system.  It was my first backup system I ever used for a different job before I started our company.  They had Amanda in-house.  It was a visual effects company.  So they had tons and tons of data to back up for all the various movies that they were  working on.  So it was really important to them.  And one of the things I really liked about it is due to native tool usage in Amanda,  we were able to do things like recover from a very painful experience from one of our  system and accidentally deleted the index directory of Amanda.  So Amanda has an index that maintains.  But if you lose that, you are not actually dead in the water.  You just use tar and dump and you are on your way.  I mean, it was painful, but we didn't lose any.  There was no data loss, which was nice.  So Amanda did a good job of protecting our data through the years, which is why we held  on to it and used it for so long.  I think, you know, as with any first love, there is issues that kind of caused us to  go looking, you know, our wandering eye maybe.  We stopped using Amanda due to the age of our current packages we had installed.  So we were on freeBisD and things kind of kept moving.  The upgrade from Amanda 2 to Amanda 3 ran into, it was kind of rough.  There was lots of renaming, UIDs and permissions and there was various time-lots and random  hangs.  So we had a real hard migration trying to go from 2 to 3.  So basically at that point we decided we needed to look for something that would be more stable  and could grow with our organization.  Another issue was that at the time, Amanda didn't support ZFS very well.  And so we were very excited about using ZFS right from the beginning.  I think since FreeBisD, it was available on 8 and we started using it like right away,  even before it was kind of the standard thing you would even use as a file system on FreeBisD.  And because of that, Amanda didn't work well with it to begin with.  And Amanda also would commonly, one of the problems we would run into is it would commonly  bring down some machines if you had rather large backups.  So if you don't know much about Amanda, but it'll spool up a lot of data into temporary  directories.  So if it was short on disk space, you could guarantee bringing a machine down with Amanda  trying to back it up, which is unfortunate.  And feel free to ask questions along the way about any of our experiences with these tools.  The next one I actually looked at was Bacula.  Well, actually we did use.  We kind of proof of concepted it and therefore about a year.  And then when we finally got all those troubles with Amanda built up, we basically said we  had to make a change and move into Bacula.  Actually Dan was probably one of the reasons I went to Bacula because of all those great  posts on the FreeBisD diary about using Bacula.  So I had a great guide to get us into a good spot with Bacula.  The back of this file system level backup, it used to be a GPLv2 and I don't think things  improved by moving to the AGPLv3 or you can get a proprietary license.  And I'll go into that a little more later.  I think this should have been telling right off the bat.  When you come to the Bacula website, they don't necessarily encourage you to use Bacula  right away.  This is right on the front page.  I think Bacula is not a simple system.  As you kind of saw in this, this is actually like an architecture diagram of how Bacula  works on this screen right here.  And there are lots and lots and lots of moving parts to Bacula.  So if you are not used to lots and lots of moving parts or if you just have a very simple  set of needs, Bacula is definitely not the choice for you.  Now we were, we did have an infrastructure of a good enough size I think it deemed worthy  of Bacula and actually Bacula's service very, very well.  Again, restoring customer mistakes since 1999 and Bacula has helped us recover from many  of them.  But there's not, not without, Bacula has not been without some drama.  Maybe not the drama like we saw earlier, but there's definitely been some friction with  the Bacula community and the Bacula creator and what he called the hidden cost of creating  open source.  So Bacula moved to an open core model.  There's an enterprise version of it which gives you the proprietary license.  And one of the promises that they made was to any feature they include in Bacula Enterprise  within some amount of time, a maximum of five year delay, would land then in the Bacula  open source product.  That hasn't always happened.  For example, one of the main things we would like to have had in Bacula would be block  level backups which is included in their open source version 5.1 which was released in 2011.  And it's still not actually in the open source version as of today.  So that, it just, this is kind of a bad example of open core gone wrong.  And so I'm not a big fan of that.  I definitely want to have more of an open source flexible solution where I feel like  it's kind of a partnership between me and the community who's, you know, producing the  open source software.  So beyond the political issues of Bacula, and Bacula is a very good product, but I think  maybe we had some issues with the, our use cases for how we backed up our data and the  kind of workloads we were backing up specifically.  One of the problems that we've run into is Bacula is, because of the various moving parts,  there's lots of configurations.  I'll show you some examples of how we improved upon that.  But there's many, many configurations that at some point our configs got so bad and so  un-maintained that we were basically down to having a once a week backup of all of our  stuff because it was, things had just gotten out of whack.  Bacula does require a database, so there's many moving parts.  So the database is there to index the files that all get backed up, which is nice when  going to restore because it's very fast to like get listings and find the files you need  to restore.  And in that case, you know, Bacula is very, very nice.  But it takes lots of care and feeding to make sure that things are always running very smoothly.  Now, in our case, we run various web applications and the database that backed those web applications,  if they, they never use Zope in the room.  I got a few people.  Okay, so the default storage for Zope is a large file, system-based file, and that file  can get very big when you go to file, file level backups.  That whole file changes every day.  Actually, it's changing every time the transaction happens in the database.  So for us, every night's backups of any incrementals would always include the largest file in the  whole system, which is typically that big database file, which is paying the butt.  It means for very inefficient use of our disk space for the backups and very slow for reading  and writing that, that giant file, maybe only a couple hundred K of that file may have even  changed.  But then we're backing up a whole many, many, maybe tens, if not hundreds of gigabyte single  file.  That's not to say that our application was perfect and could have been improved fun,  but because of our specific use cases, it was very, very slow.  Yeah, so with the slow to backup part, it does leverage a file daemon on every box.  So if you're running back-in, you will install a file daemon on every client machine that  you'd want to back up.  And then there will be a director piece that actually tells a backup storage daemon to  stream the data from the file daemon on the client back to the backup server or wherever  you're storing, wherever you're either to tape or to some kind of disk storage.  And to actually help configure our bacula, we ended up creating sets of Python code to  produce templates and actually would run a set of scripts to generate our many thousands  of lines of bacula configuration each time a change had to be made.  So I think that there has to be a better way.  Oh, funny enough, there might be a better way.  What if, so this is like when you start thinking outside the box and actually looking at what  you really need for a backup solution, you have to kind of analyze the various options  that are out there and find one that fits your specific needs best.  And for us, because of this web application that we typically develop or deploy to, how  having a block level backup is pretty critical to us being able to do efficient and fast  backups and making sure that they always happen.  But what if there was a file system that supported some of these advanced features we were looking  for?  Now, for example, snapshots, replication, compression, kind of built in data integrity.  I think something like this exists, which is exactly what we were looking for.  And also, what if there was a way for all of our VMs and all of our jails to be backed  up by a file system that supported that block level changes and incremental snapshotting  and replication?  All right, and did that at a block level.  Did I mention block level?  That's really important to us.  Given that we were a FreeBSD and we love ZFS, this is a natural fit.  So we have been using FreeBSD and it was nice is that ZFS actually is cross-platform.  So depending on whether we had maybe a Solaris, or actually at some point we did have Smart  OS, Open Solaris, FreeBSD.  I don't think we ever had Linux running with ZFS, but that's actually quite stable now  from some of the work that's been done.  But it's nice because then it works cross-platform.  We can actually use the same solution across all those various OSes.  And actually we were able to greatly simplify how we were doing our backups.  For the goals, and these kind of just apply anytime you're setting up and trying to figure  out how to accomplish your backup tasks, is it has to be reliable.  We have to make sure that we can ensure that it runs and if there's errors in the process  that we get, obviously a report or someone gets alerted to go fix the issues that are  happening.  It needs to be granular.  We need to be able to get down to a specific VM or a specific file directory and be able  to restore a file quickly from that backup, especially if it's that large database file.  Now we really care about integrity.  How many people who have been on a Mac for any amount of time have run into the wonderful  HFS plus silent file bit rot?  Yeah.  This has bitten me.  I'm actually running, you know, prior to the high Sierra, ZFS on my laptop, exactly  for those reasons.  So I use Mac, or open ZFS on Mac.  I've been very happy with it.  And the integrity issues are awesome.  So I kind of use it on my Mac.  We also use it in all of our infrastructure.  Another key piece here is having few dependencies.  I want as little as possible to have to touch the VMs or the jails.  They know as little about the backup process than they need to.  And it needs to be easy to configure.  And then obviously at the very end it runs with little care and feeding.  So when moving over to using ZFS for our backup solution, the first and initial win we got  was obviously going to be ease of use and simplicity of configuration.  So this is on our backula setup.  These are the amount of configuration lines that are required to backup.  I think we've got, probably, like 300 hosts or 200 hosts inside this.  And you can see here we're well over 1,000 lines of configuration to manage that backup.  And if you weren't using some kind of automated scripting to produce this, this is very tedious  and very possible, very error prone, and which led into our issues with backula being the  config getting out of date and us basically getting down to like one backup a week.  So every server that has to be backed up will have to have the backula fd process.  So every server has this 27 line configuration file.  And then the director has the others.  You can see there the regular, like scheduler and jobs files.  So we had to have, again, some way of managing this process and we had written a bunch of  custom Python to be able to do this for us.  But again, that falls out of repair, disrepair quickly.  When moving into using ZFS for our new backup solution, instead of having every specific,  every VM and every jail running the backula file daemon, we changed our, shifted how we  did that.  Instead of doing it at the client level, every client backing up, we then took every jail  host and every beehive host and actually backed them up at the hypervisor or jail host level.  Which again greatly simplifies things because you may have 30, 40 jails running on one machine.  You may have 10 to 15 virtual machines on one machine.  Now we're down to configuring one set of configurations instead of like 15 or 30 per  machine, which is nice.  So this is on the jail host, for example.  We use a combination of ZFS SNAP and ZXFOR to do the snapshot management and then the  replication.  Now what's nice about ZFS SNAP and those tools specifically is that they are, they have very  few dependencies.  Most of them are just shell scripts that work cross-platform.  The ZFS SNAP works on my Mac, just as well as it works on previous D, which is nice.  But to enable the snapshot management, you just have to add a couple of lines to your  periodic.conf and from there you basically enable it, tell it which recursive file systems  you want to grab and then it manages all from there.  This configures daily and weekly.  You can actually go down to other granularities, but for us that was enough to satisfy our  promise to our customers with how much we would retain for them, for their data usage.  And so that's on the, so every, every jail, every B-hype host has this in it now.  And then on every trusted server, so like for example the hyper-riser servers are considered  trusted because we don't have clients logging into them, we would use ZXFOR to send back  the incremental snapshots.  So in CronTab there's one line and that handles sending all the snapshots.  So ZSnap, ZFS snapshots creates the snapshots, ZXFOR on a once a day process will send them  all back to our Freeness servers.  So they're all running jail or we're all running the Freeness for our storage architecture  on the back end and I'll show you some cool stuff we've done with the Freeness stuff later.  But what's nice here, this is pretty easy to configure.  Alan is the kind of somewhat maintainer of ZXFOR but it doesn't require a lot.  It's very simple, very straightforward.  You know, it can delete snapshots on the destination, it can force rollbacks of file systems in  case it wasn't kind of ready to receive the snapshots for some reason.  What else is here, it's interesting.  And then the recursive data sets.  So yeah, one line, we're sending them back everything to the servers.  Now any server where we didn't necessarily trust the host, like maybe we did have some  bare metal virtual or bare metal hardware servers hosted for customers, we don't want  to put our keys on that host that could access our storage server.  So instead of we kind of flip the direction, we would pull from that bare metal host back  to the Freeness server instead of pushing from the servers back into Freeness.  So in this case, it's just a simple SSH send receive SSH into the remote host and then  send the receive back into the Freeness server.  And that was just mostly because we didn't want to have our potentially private keys  on some remote, who knows what server.  So this is on the Freeness side of things, we basically have a cron job that will run  that, this script right here.  So we just basically use the scheduling, you know, right through the GUI so any admin  can very easily set up and add in new machines if they need to or manage the very scheduled  jobs that are inside of Freeness.  Now what's nice is you get a bonus if you're using Freeness.  She's running Freeness in production right now.  Okay, so you guys all should be well aware of this.  The Freeness has, I mean, we have multiple data center locations and it's very easy to  do offsite backups.  So if you guys haven't done this, you definitely should check out the replication tasks area  of Freeness.  Basically, it's super easy to now set up, remoting that data offsite to another Freeness  machine and running in some other data center.  All with, you know, one line through the GUI, you basically set up a schedule and point  it at a data set and it can now send those things offsite.  So it's kind of an added benefit to certain customers who want to kind of upsell into  offsite backups of their data.  It's very easy for us to just now come in and add one line per customer who wants to  have that offsite feature enabled.  Then very easy for the admins to manage and maintain.  So this presentation is online in my GitHub, my email and my Twitter.  So if anybody's had any questions, I am happy to answer them.  No questions?  All right, Dan.  Instead of time-dropped, have you considered, let's say, periodic?  Yes.  So we do for part of it using the ZFS snap stuff, but I'm not actually like, you know,  formalize that more.  The reason I like periodic is it can be dispelled by a fork or something, and you're not messing  around with a cron-tap, but it's just a disk.  I know the cron-tap is a fist bump.  And you can also enable and disable it with nrc.com.  True.  Yeah, so I agree.  That's definitely a better way to go to add new periodics for the scheduling.  If I can cron.d?  Yeah.  So we use, and so all of our servers are configuration managed by salt, and so we actually just  have that as part of the default, you know, salt profile when a machine's brought up,  this stuff just gets put in place automatically.  Like, luckily there's no humans involved typically in getting any of this running.  So new machines are automatically snapshotting and sending data back in day one.  Could you give us an idea of the improvement in both the time performance and backups  on the storage?  Oh, yeah.  I probably didn't mention that in my notes here.  The back gala backups were getting to a point where they were taking longer than 24 hours  sometimes.  So we would have trouble basically doing them daily.  That's what led us to the kind of bad situation.  We did get it down to, say, like, 14 or 15 hours before we switch over to ZFS.  And when the switch over to ZFS, it now back up the same set of servers.  Once you've got the initial base backups and all the base snapshots sent over to the  storage server, the new runs only take a couple hours, like maybe an hour or two, to run through  all of our infrastructure and send the incremental snapshots.  So it's tremendously faster and saves a lot more space on data.  A little bit of a question.  All right.  How much, can you say, a couple hours to transfer to do the replication?  How much bandwidth do you have between the data centers?  Oh, for us, between the data centers, it's 100 megaline.  But not every data set gets sent across to the remote data center.  We only offsite for customers who want that extra feature and then also just for our stuff.  Go ahead.  When you were, when you were generally unhappy with various, one of the reasons that you  were unhappy with was the commercial to community relationship.  Did you guys look at various at all?  We did not.  I'm not actually not even familiar with that.  Is that the commercial?  Okay.  Because of this.  I mean, again, for us, the simplicity, it was a bigger win than almost the politics and  the commercial licensing pieces of it.  Luckily, we don't have to restore very often.  But the restores are easy.  You go into either the machine and send receive something back or typically it's one file.  Somebody's like, I messed up a single file.  And so we can actually from the free now machine just log in CD into the specific snapshot  directory and grab the file, you know, write a file system.  So in other words, the snapshots you send, you basically fully replicate file system on here?  Correct.  Now, we have to get the clone and mount that clone if you want to like go in and get the  data.  But yeah, we do.  It's all of us just all I'll put back on the freedom system.  What do you do about picking up old snapshots?  So the old snapshots get cleaned up automatically from the zx for command.  It will delete.  Well, as the ZFS snap takes the snapshots, it cleans them up on the client's side.  And then on the server side, the zx for deletes them if they don't exist on the destination.  So from the client to the free now box.  So one of the keeping library of old ones in case, you know, a file gets deleted and  that machine gets copied over the snapshot.  Right.  So we keep we were keeping a month for a year and then weeklies for I can't remember what  the time frame was.  That's all configurable be the ZFS snap.  Yeah  Yeah  Right  Yeah  Yeah  Right  Right  Yeah, probably my biggest issue with, you know, snapshot management is the fact that  ZFS is too good and it's hard to delete data if I needed to because of, like, if there's  some kind of legality and someone required me to delete some kind of backup, I'm going  to have a hard time actually with that.  Yeah.  Well, and I liked your suggestion during the BOP or the ZFS talk today was to encrypt  those data sets, have encrypted data sets and so basically the deletion process is throw  away that key.  And I think that's a good idea for now.  So one of my big fears with replicating snapshots where you propagate the deletion  from the source, there's been cases where I accidentally deleted the snapshots on the  source and I wouldn't want those populated to the backup.  If they disappear, I'm probably going to have to fix quite some of that as a grandfather  in the game there, but I would rather have a minimum sort of time before things go away  as well.  Yeah, actually, I would like to keep, I would like to guarantee keeping at least the last  24 hours with in case a certain idiot.  That's a good idea.  It should be pretty easy to add.  Yeah.  Although the next talk is...  I was going to say the next talk, I think, is really, made me really relevant.  It works very well, I must admit.  I need to write it.  I just need to make sure.  Yeah.  Don't be too too...  I'm on the domain for a rewrite.  You never started.  No, the next...  Actually, I talked to the next speaker about his Z-repl and it seems to be a much more  formalized system around the snapshot and replication management piece, which I wasn't...  It's brand new and he's kind of the only one using it right now, but I was very interested  in actually picking that up and starting to use it as well because I think it may complement  our set of tools really well.  And now, the only thing I don't like about it is it adds in more complexity.  This is already very, very simple and does accomplish the job, but if I can improve on  it by...  And the value is there in that improvement, then it's worth doing.  Yeah, question.  Yeah, so the question is how do we go about protecting against, say, a rogue system in  or some kind of human error with an offline copy of the data?  Right now, we just have the off-site, but it's not offline.  That could very easily propagate some fat fingering across to the disaster recovery center.  Yeah, there's the two cases.  Something blows up that's fine, but malicious or accidents is a real concern as well.  Yeah, I don't currently have an offline solution for that right now.  I mean, one of the things we do...  Yeah.  I mean, for us, it is more important that we have the off-site had more up-to-date data  though, too.  Yeah.  Yeah.  Well, that's something we've thought about at the off-site center having a second Frenez  box there that they do...  The replica drops the data down, like, say, on a very specific schedule, so you have that  protection of an additional copy there.  Have you or anyone here looked at, say, two-factor authentication or even secure levels to prevent  the pollution of snapshots?  I haven't.  And that's actually one of the things that I...  Like, one of them...  I don't know, not a problem, but like, one of my issues with ZFS is it's very easy and  very quick to destroy.  There's not even an ask you, like, if you care to destroy it.  It's just...  You've just destroyed it and it's very efficient.  Thank you, Matt.  But...  I should almost...  At least that on every machine, then.  I could say for many times by the back, I almost always typo destroy.  No one ever alias ZFSD story to destroy.  But yeah, it is true.  It's very easy to destroy things on ZFS.  I have not looked, but once you destroy it, there's not like a, I can get it back type  thing.  It's gone.  The snapshots, yeah, it's true.  Which I have lots of snapshots, which is very nice.  Yeah.  So I'm going to say, how are you prepared to deal with, say, if a core telling you that  a certain set of data must be destroyed from...  Well, I'm saying it.  I'm not at this point.  It's very, very difficult to get rid of data in this setup.  Luckily, you know, cross fingers.  It literally happens better.  Right?  Yeah.  So I don't even know, like, how we could go about that.  You need to have a separate data set for every unit of data.  Like, it took us a while to realize, but one data set for every single customer.  Which is what we have.  We do have one data set for every individual customer, because every customer has either  a VM or a jail, and all the jails are, each of them have their own data sets.  So a specific customer came to us and said, we need to have our data deleted, and yes,  they can destroy all those data related to just their data.  If they have a subset of their data, that's the only option to make them do VM.  How can you get the data out?  Right.  That's probably exactly what I'd have to do, because it's just the data stays around.  That, it has to protect you.  You don't want it to.  Yeah.  Then you can put data sets.  Yeah.  I mean, the data sets are cheap, so use them, you know, all over the places, which is nice.  Yeah.  So, the way that I clean out unwanted data from backup snapshots is when I make a writable  clone of the snapshot, then I eliminate what I don't want in there.  You have to do that for every snapshot where the data is, and then you delete the original  snapshot, and then rename, resulting snapshot back to the original name, then you actually  have the right snapshot with the correct names.  Yeah.  So, you're saying it's possible to make a writable clone, delete the file, but you have  to do this for every snapshot you would have.  Could be, you know, we potentially have hundreds or thousands of snapshots.  I would say that sounds like it's calling out screaming for automation.  Yeah.  That's actually one of my favorite features of ZFS is the.CFS directory.  Yes.  So, I don't know if you'll have an answer for this, but there's been very good news  in the town.  I've got some data where I have either contractual or regulatory requirements to put it on offline  media, whether it be Word media or tape.  Yes, I still do the tape world.  How do you get ZFS data onto the tape efficiently nowadays?  So, the ZFS data is a file, it's a serialized property.  You can put it on tape and then receive it a year later.  So, release ZFS and redirect to.CFS.  Oh, pipe it to tar.  Yeah.  So, the question is how do you get ZFS data sets down to tape, and you can just stream  it down to tape, ascend.  Do you do the full under tape and you put it on the end of the issue?  That's too easy.  How would you like to set it apart?  You know what tar stands for, right?  Yes, that's true.  ZFS has made our lives too easy.  It's Michael.  Are you using specifically LZ4, sorry, LZ4, and have you ever attempted deduplication  that hasn't suited you?  I've never attempted deduplication.  We've had enough storage.  That's not an issue.  We are using LZ4.  Nothing unusual?  Nothing, I've never noticed.  No.  Is there something I should be aware of with LZ4?  Yeah, there's something you're coming about.  Well, I'm looking forward to the new stuff Alan is talking about.  Yeah, that I'm actually quite excited about.  So I'm happy to switch when the time comes.  Yes.  Have you given any issues?  It sounds like a Zofi since most of the flat file data is.  Have you given any cases where there's a process that's caching things heavily into memory  and doesn't flush as often as it should?  Where I mean to talk something like that.  So the question is if we run into an issue where something is maybe not flushing down  the disk as aggressively as we'd like with the application.  It's not actually an issue with Zope.  I mean now we're getting more into a Zope discussion specifically.  Zope is an append-only database and it on start will automatically trim any corrupted  transactions from the end of the database as it starts.  So you can very reliably snapshot that file at any point in time.  Even if the power outage happened or whatever it's very, very resilient to starting back up  and basically rolling back whatever failed transactions might have been on the end of it.  So for that we've just benefited because that's the nature of that application.  If you have other databases where you want to ensure consistency like Postgres or MySQL  I mean you should definitely have probably an additional cron jobs and in those cron jobs  maybe to do some kind of a start backup, stop backup type thing.  One more question and this is probably for Alan.  Is there a way we can do this backup to force or if you have a ZIL to force it to flush?  That I don't know.  I think it's definitely a question for the right in front of you.  What you're saying is the snapshot will be the same as the created one.  Okay so the question is about the ZIL flushing on, but it happens automatically when you do a snapshot.  And since you only send snapshots you have the whole data set.  Always be consistent. I love that.  Just to combine a couple of recent questions, slightly off topic data, but there's like an extra strategy for when they have backups on offline media like pay for CDs.  And then they get a CDPR request telling them they need to remove all data related to the typical customer.  I've only had a request like that once where we had customer data we shouldn't have had and it was in subversion.  And so I had to filter it out of every subversion transaction to undo it, but I didn't have it on tape.  We probably did, but we're using backill at the time.  Yeah.  Backill would have attended to that for you.  So basically some of the data that I have to put onto archival media is government data. We have an unofficial opinion that as long as it is right only media, so word media, whether LTO or MDS, or anything like that, as long as it is not ultra bulk, the answer legally can be no, we can't do that.  If you are able to offer the media, you are allowed to say no, we can't do that.  So you choose your media carefully.  This is yet to be tested. This is an unofficial opinion that we've been told verbally, but that's something you might explore through your legal team.  Because almost all the legislation around the world allowing this does have some exemptions for where it is not technically possible to delete the data.  And G8 is on optical media or something like that.  Yeah.  Couldn't you get into like a semantics thing if you have a check some stream?  Absolutely.  That's not changed.  That helps.  That'll be next year's presentation.  You talk to your legal team, etc.  But yeah, there are ways to do this that don't put yourself in the foot.  And that's like the logical stream of like one database for each person.  One database for every individual in Europe.  Yeah.  I don't know if your opinion law situation is, but this is under Canadian law and that's what we've been told.  Whether it's mature or not, I am not a lawyer.  Who in the room has gotten a notice to delete data?  Okay.  Coming.  Well, not under GDPR, but under pre-project.  Oh, actually, I'm sorry.  We have season this, had season this since the years when the BBC project, the man with the most diverse workflows and stuff that people really want to experience.  And do you have to go, you've gone all the way back through the backups of that?  In that case, at the time, it was what backups, but...  It was in CVSAR.  Yeah.  Globally distributed backups at the time.  That's the official story.  I'm sorry, the official line.  So they do happen and yeah.  I expected to get formal common works to figure out how GDPR is intended to be used.  Yeah, this, I mean, I'm kind of worried about this because we do write applications that are portals where people log in and have a profile.  And I don't know how to reliably say I've gotten rid of all your data.  Like, there'll be these remnants.  Or I can say, I don't know, with GDPR, do you have the time frame with which you have to guarantee it's all gone?  I think the general rule is if you can make a case that what you're doing is very reasonable, they're more interested in the fact that you can do it and you're doing the right thing than specific doting guys and possibilities and specific timelines.  Yeah.  So you have a clause about, has to be commercially reasonable and you can't bankrupt yourself to leave the data.  It's complicated not a lawyer.  There's a lot of good write-ups on it.  We do have a little case in talk to a lawyer.  Discussion.  Talk to a lawyer.  Yeah.  So talk to a lawyer is our answer.  Almost makes you wonder if they didn't make this happen.  Any other questions?  Thank you all very much.
Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_0_107.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_108_251.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_252_515.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_516_2099.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_2100_2759.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_2760_3623.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_3624_4151.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_4152_4199.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_4200_4295.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_4296_4811.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_4812_5051.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_5052_5195.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_5196_5339.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_5340_5519.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_5520_5711.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_5712_6815.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_6816_7055.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_7056_7703.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_7704_8819.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_8820_9299.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_9300_9515.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_9516_10175.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_10176_10955.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_10956_11939.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_11940_16259.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_16260_16319.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_16320_16823.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_16824_17759.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_17760_17879.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_17880_18047.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_18048_18839.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_18840_19367.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_19368_19607.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_19608_19883.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_19884_20291.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_20292_20963.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_20964_21071.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_21072_22619.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_22620_25055.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_25056_25211.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_25212_25295.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_25296_26939.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_26940_27371.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_27372_28091.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_28092_28427.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_28428_28475.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_28476_28823.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_28824_29159.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_29160_29603.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_29604_30239.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_30240_37655.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_37656_37703.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_37704_64271.png Calvin_Hendryx-Parker__Replacing_Traditional_Backup_Systems_with_ZFS_64272_64302.png
Welcome you all to my little talk about replacing traditional backup systems with ZFS. My name is Calvin Hendricks Parker and I work for a company in Indiana called Six Feet Up. We have been around since 1999, which is kind of crazy to say out loud because it means next year is our 20th year in business, which is crazy. But the whole time has been on previous team. I'm a long time previous user. I've been using it since 1996 and has kept our company safe, our data safe. And I want to talk about some of the journey that we have done over the years, you know, through various backup systems and how we have backed up our data. But basically since 1999, we've run a cage, an or a rack or host data for others someplace. So we have to keep that data safe. And we've been successfully restoring customer mistakes ever since then. Good job on gender diversity. Oh yeah, we are definitely an anomaly. We are a woman-owned business. This is the CEO right here. Also my wife, very wonderful person, I had to say that. But yeah, we've done a good job of trying to be a diverse company. So if you're looking for a job, we are hiring, looking for a developer right now. So I think the key to all of this is backups will be done when it is easy. I said by every system in the world. More importantly, like simple. I think easy should probably be taken out of my vocabulary. I know that my system is cringe whenever I use that word. So if you consider a bad word and really should make sure that backups are as simple to accomplish as possible because then they'll be done on a regular basis. So a little bit of backstory of where we have been with backups. We started from year one with backups like any good company should, even a small company like ours. We're only 16 people right now. When we first started, it was just actually my wife Gabrielle in 1999. And so I started with Amanda as a backup system from 99 to 2012. We did a little bit of Bacula in 2007 as a proof of concept. And then dofolian on Bacula in 2012. And I'll talk a little bit about each of those systems. Has anyone in the room been using Amanda? Yeah, this is like. Yeah, Amanda is, I guess, the old standard of backup systems and open source. If you've not used it, it is pretty awesome because it is very easy to use. Some of the things I really loved about it is written in C in Pearl, has a BSD style and GPL license. It is currently backed and supported by a commercial entity. So there is people actively working on it. I would recommend it to people because it is still a good backup system. It was my first backup system I ever used for a different job before I started our company. They had Amanda in-house. It was a visual effects company. So they had tons and tons of data to back up for all the various movies that they were working on. So it was really important to them. And one of the things I really liked about it is due to native tool usage in Amanda, we were able to do things like recover from a very painful experience from one of our system and accidentally deleted the index directory of Amanda. So Amanda has an index that maintains. But if you lose that, you are not actually dead in the water. You just use tar and dump and you are on your way. I mean, it was painful, but we didn't lose any. There was no data loss, which was nice. So Amanda did a good job of protecting our data through the years, which is why we held on to it and used it for so long. I think, you know, as with any first love, there is issues that kind of caused us to go looking, you know, our wandering eye maybe. We stopped using Amanda due to the age of our current packages we had installed. So we were on freeBisD and things kind of kept moving. The upgrade from Amanda 2 to Amanda 3 ran into, it was kind of rough. There was lots of renaming, UIDs and permissions and there was various time-lots and random hangs. So we had a real hard migration trying to go from 2 to 3. So basically at that point we decided we needed to look for something that would be more stable and could grow with our organization. Another issue was that at the time, Amanda didn't support ZFS very well. And so we were very excited about using ZFS right from the beginning. I think since FreeBisD, it was available on 8 and we started using it like right away, even before it was kind of the standard thing you would even use as a file system on FreeBisD. And because of that, Amanda didn't work well with it to begin with. And Amanda also would commonly, one of the problems we would run into is it would commonly bring down some machines if you had rather large backups. So if you don't know much about Amanda, but it'll spool up a lot of data into temporary directories. So if it was short on disk space, you could guarantee bringing a machine down with Amanda trying to back it up, which is unfortunate. And feel free to ask questions along the way about any of our experiences with these tools. The next one I actually looked at was Bacula. Well, actually we did use. We kind of proof of concepted it and therefore about a year. And then when we finally got all those troubles with Amanda built up, we basically said we had to make a change and move into Bacula. Actually Dan was probably one of the reasons I went to Bacula because of all those great posts on the FreeBisD diary about using Bacula. So I had a great guide to get us into a good spot with Bacula. The back of this file system level backup, it used to be a GPLv2 and I don't think things improved by moving to the AGPLv3 or you can get a proprietary license. And I'll go into that a little more later. I think this should have been telling right off the bat. When you come to the Bacula website, they don't necessarily encourage you to use Bacula right away. This is right on the front page. I think Bacula is not a simple system. As you kind of saw in this, this is actually like an architecture diagram of how Bacula works on this screen right here. And there are lots and lots and lots of moving parts to Bacula. So if you are not used to lots and lots of moving parts or if you just have a very simple set of needs, Bacula is definitely not the choice for you. Now we were, we did have an infrastructure of a good enough size I think it deemed worthy of Bacula and actually Bacula's service very, very well. Again, restoring customer mistakes since 1999 and Bacula has helped us recover from many of them. But there's not, not without, Bacula has not been without some drama. Maybe not the drama like we saw earlier, but there's definitely been some friction with the Bacula community and the Bacula creator and what he called the hidden cost of creating open source. So Bacula moved to an open core model. There's an enterprise version of it which gives you the proprietary license. And one of the promises that they made was to any feature they include in Bacula Enterprise within some amount of time, a maximum of five year delay, would land then in the Bacula open source product. That hasn't always happened. For example, one of the main things we would like to have had in Bacula would be block level backups which is included in their open source version 5.1 which was released in 2011. And it's still not actually in the open source version as of today. So that, it just, this is kind of a bad example of open core gone wrong. And so I'm not a big fan of that. I definitely want to have more of an open source flexible solution where I feel like it's kind of a partnership between me and the community who's, you know, producing the open source software. So beyond the political issues of Bacula, and Bacula is a very good product, but I think maybe we had some issues with the, our use cases for how we backed up our data and the kind of workloads we were backing up specifically. One of the problems that we've run into is Bacula is, because of the various moving parts, there's lots of configurations. I'll show you some examples of how we improved upon that. But there's many, many configurations that at some point our configs got so bad and so un-maintained that we were basically down to having a once a week backup of all of our stuff because it was, things had just gotten out of whack. Bacula does require a database, so there's many moving parts. So the database is there to index the files that all get backed up, which is nice when going to restore because it's very fast to like get listings and find the files you need to restore. And in that case, you know, Bacula is very, very nice. But it takes lots of care and feeding to make sure that things are always running very smoothly. Now, in our case, we run various web applications and the database that backed those web applications, if they, they never use Zope in the room. I got a few people. Okay, so the default storage for Zope is a large file, system-based file, and that file can get very big when you go to file, file level backups. That whole file changes every day. Actually, it's changing every time the transaction happens in the database. So for us, every night's backups of any incrementals would always include the largest file in the whole system, which is typically that big database file, which is paying the butt. It means for very inefficient use of our disk space for the backups and very slow for reading and writing that, that giant file, maybe only a couple hundred K of that file may have even changed. But then we're backing up a whole many, many, maybe tens, if not hundreds of gigabyte single file. That's not to say that our application was perfect and could have been improved fun, but because of our specific use cases, it was very, very slow. Yeah, so with the slow to backup part, it does leverage a file daemon on every box. So if you're running back-in, you will install a file daemon on every client machine that you'd want to back up. And then there will be a director piece that actually tells a backup storage daemon to stream the data from the file daemon on the client back to the backup server or wherever you're storing, wherever you're either to tape or to some kind of disk storage. And to actually help configure our bacula, we ended up creating sets of Python code to produce templates and actually would run a set of scripts to generate our many thousands of lines of bacula configuration each time a change had to be made. So I think that there has to be a better way. Oh, funny enough, there might be a better way. What if, so this is like when you start thinking outside the box and actually looking at what you really need for a backup solution, you have to kind of analyze the various options that are out there and find one that fits your specific needs best. And for us, because of this web application that we typically develop or deploy to, how having a block level backup is pretty critical to us being able to do efficient and fast backups and making sure that they always happen. But what if there was a file system that supported some of these advanced features we were looking for? Now, for example, snapshots, replication, compression, kind of built in data integrity. I think something like this exists, which is exactly what we were looking for. And also, what if there was a way for all of our VMs and all of our jails to be backed up by a file system that supported that block level changes and incremental snapshotting and replication? All right, and did that at a block level. Did I mention block level? That's really important to us. Given that we were a FreeBSD and we love ZFS, this is a natural fit. So we have been using FreeBSD and it was nice is that ZFS actually is cross-platform. So depending on whether we had maybe a Solaris, or actually at some point we did have Smart OS, Open Solaris, FreeBSD. I don't think we ever had Linux running with ZFS, but that's actually quite stable now from some of the work that's been done. But it's nice because then it works cross-platform. We can actually use the same solution across all those various OSes. And actually we were able to greatly simplify how we were doing our backups. For the goals, and these kind of just apply anytime you're setting up and trying to figure out how to accomplish your backup tasks, is it has to be reliable. We have to make sure that we can ensure that it runs and if there's errors in the process that we get, obviously a report or someone gets alerted to go fix the issues that are happening. It needs to be granular. We need to be able to get down to a specific VM or a specific file directory and be able to restore a file quickly from that backup, especially if it's that large database file. Now we really care about integrity. How many people who have been on a Mac for any amount of time have run into the wonderful HFS plus silent file bit rot? Yeah. This has bitten me. I'm actually running, you know, prior to the high Sierra, ZFS on my laptop, exactly for those reasons. So I use Mac, or open ZFS on Mac. I've been very happy with it. And the integrity issues are awesome. So I kind of use it on my Mac. We also use it in all of our infrastructure. Another key piece here is having few dependencies. I want as little as possible to have to touch the VMs or the jails. They know as little about the backup process than they need to. And it needs to be easy to configure. And then obviously at the very end it runs with little care and feeding. So when moving over to using ZFS for our backup solution, the first and initial win we got was obviously going to be ease of use and simplicity of configuration. So this is on our backula setup. These are the amount of configuration lines that are required to backup. I think we've got, probably, like 300 hosts or 200 hosts inside this. And you can see here we're well over 1,000 lines of configuration to manage that backup. And if you weren't using some kind of automated scripting to produce this, this is very tedious and very possible, very error prone, and which led into our issues with backula being the config getting out of date and us basically getting down to like one backup a week. So every server that has to be backed up will have to have the backula fd process. So every server has this 27 line configuration file. And then the director has the others. You can see there the regular, like scheduler and jobs files. So we had to have, again, some way of managing this process and we had written a bunch of custom Python to be able to do this for us. But again, that falls out of repair, disrepair quickly. When moving into using ZFS for our new backup solution, instead of having every specific, every VM and every jail running the backula file daemon, we changed our, shifted how we did that. Instead of doing it at the client level, every client backing up, we then took every jail host and every beehive host and actually backed them up at the hypervisor or jail host level. Which again greatly simplifies things because you may have 30, 40 jails running on one machine. You may have 10 to 15 virtual machines on one machine. Now we're down to configuring one set of configurations instead of like 15 or 30 per machine, which is nice. So this is on the jail host, for example. We use a combination of ZFS SNAP and ZXFOR to do the snapshot management and then the replication. Now what's nice about ZFS SNAP and those tools specifically is that they are, they have very few dependencies. Most of them are just shell scripts that work cross-platform. The ZFS SNAP works on my Mac, just as well as it works on previous D, which is nice. But to enable the snapshot management, you just have to add a couple of lines to your periodic.conf and from there you basically enable it, tell it which recursive file systems you want to grab and then it manages all from there. This configures daily and weekly. You can actually go down to other granularities, but for us that was enough to satisfy our promise to our customers with how much we would retain for them, for their data usage. And so that's on the, so every, every jail, every B-hype host has this in it now. And then on every trusted server, so like for example the hyper-riser servers are considered trusted because we don't have clients logging into them, we would use ZXFOR to send back the incremental snapshots. So in CronTab there's one line and that handles sending all the snapshots. So ZSnap, ZFS snapshots creates the snapshots, ZXFOR on a once a day process will send them all back to our Freeness servers. So they're all running jail or we're all running the Freeness for our storage architecture on the back end and I'll show you some cool stuff we've done with the Freeness stuff later. But what's nice here, this is pretty easy to configure. Alan is the kind of somewhat maintainer of ZXFOR but it doesn't require a lot. It's very simple, very straightforward. You know, it can delete snapshots on the destination, it can force rollbacks of file systems in case it wasn't kind of ready to receive the snapshots for some reason. What else is here, it's interesting. And then the recursive data sets. So yeah, one line, we're sending them back everything to the servers. Now any server where we didn't necessarily trust the host, like maybe we did have some bare metal virtual or bare metal hardware servers hosted for customers, we don't want to put our keys on that host that could access our storage server. So instead of we kind of flip the direction, we would pull from that bare metal host back to the Freeness server instead of pushing from the servers back into Freeness. So in this case, it's just a simple SSH send receive SSH into the remote host and then send the receive back into the Freeness server. And that was just mostly because we didn't want to have our potentially private keys on some remote, who knows what server. So this is on the Freeness side of things, we basically have a cron job that will run that, this script right here. So we just basically use the scheduling, you know, right through the GUI so any admin can very easily set up and add in new machines if they need to or manage the very scheduled jobs that are inside of Freeness. Now what's nice is you get a bonus if you're using Freeness. She's running Freeness in production right now. Okay, so you guys all should be well aware of this. The Freeness has, I mean, we have multiple data center locations and it's very easy to do offsite backups. So if you guys haven't done this, you definitely should check out the replication tasks area of Freeness. Basically, it's super easy to now set up, remoting that data offsite to another Freeness machine and running in some other data center. All with, you know, one line through the GUI, you basically set up a schedule and point it at a data set and it can now send those things offsite. So it's kind of an added benefit to certain customers who want to kind of upsell into offsite backups of their data. It's very easy for us to just now come in and add one line per customer who wants to have that offsite feature enabled. Then very easy for the admins to manage and maintain. So this presentation is online in my GitHub, my email and my Twitter. So if anybody's had any questions, I am happy to answer them. No questions? All right, Dan. Instead of time-dropped, have you considered, let's say, periodic? Yes. So we do for part of it using the ZFS snap stuff, but I'm not actually like, you know, formalize that more. The reason I like periodic is it can be dispelled by a fork or something, and you're not messing around with a cron-tap, but it's just a disk. I know the cron-tap is a fist bump. And you can also enable and disable it with nrc.com. True. Yeah, so I agree. That's definitely a better way to go to add new periodics for the scheduling. If I can cron.d? Yeah. So we use, and so all of our servers are configuration managed by salt, and so we actually just have that as part of the default, you know, salt profile when a machine's brought up, this stuff just gets put in place automatically. Like, luckily there's no humans involved typically in getting any of this running. So new machines are automatically snapshotting and sending data back in day one. Could you give us an idea of the improvement in both the time performance and backups on the storage? Oh, yeah. I probably didn't mention that in my notes here. The back gala backups were getting to a point where they were taking longer than 24 hours sometimes. So we would have trouble basically doing them daily. That's what led us to the kind of bad situation. We did get it down to, say, like, 14 or 15 hours before we switch over to ZFS. And when the switch over to ZFS, it now back up the same set of servers. Once you've got the initial base backups and all the base snapshots sent over to the storage server, the new runs only take a couple hours, like maybe an hour or two, to run through all of our infrastructure and send the incremental snapshots. So it's tremendously faster and saves a lot more space on data. A little bit of a question. All right. How much, can you say, a couple hours to transfer to do the replication? How much bandwidth do you have between the data centers? Oh, for us, between the data centers, it's 100 megaline. But not every data set gets sent across to the remote data center. We only offsite for customers who want that extra feature and then also just for our stuff. Go ahead. When you were, when you were generally unhappy with various, one of the reasons that you were unhappy with was the commercial to community relationship. Did you guys look at various at all? We did not. I'm not actually not even familiar with that. Is that the commercial? Okay. Because of this. I mean, again, for us, the simplicity, it was a bigger win than almost the politics and the commercial licensing pieces of it. Luckily, we don't have to restore very often. But the restores are easy. You go into either the machine and send receive something back or typically it's one file. Somebody's like, I messed up a single file. And so we can actually from the free now machine just log in CD into the specific snapshot directory and grab the file, you know, write a file system. So in other words, the snapshots you send, you basically fully replicate file system on here? Correct. Now, we have to get the clone and mount that clone if you want to like go in and get the data. But yeah, we do. It's all of us just all I'll put back on the freedom system. What do you do about picking up old snapshots? So the old snapshots get cleaned up automatically from the zx for command. It will delete. Well, as the ZFS snap takes the snapshots, it cleans them up on the client's side. And then on the server side, the zx for deletes them if they don't exist on the destination. So from the client to the free now box. So one of the keeping library of old ones in case, you know, a file gets deleted and that machine gets copied over the snapshot. Right. So we keep we were keeping a month for a year and then weeklies for I can't remember what the time frame was. That's all configurable be the ZFS snap. Yeah Yeah Right Yeah Yeah Right Right Yeah, probably my biggest issue with, you know, snapshot management is the fact that ZFS is too good and it's hard to delete data if I needed to because of, like, if there's some kind of legality and someone required me to delete some kind of backup, I'm going to have a hard time actually with that. Yeah. Well, and I liked your suggestion during the BOP or the ZFS talk today was to encrypt those data sets, have encrypted data sets and so basically the deletion process is throw away that key. And I think that's a good idea for now. So one of my big fears with replicating snapshots where you propagate the deletion from the source, there's been cases where I accidentally deleted the snapshots on the source and I wouldn't want those populated to the backup. If they disappear, I'm probably going to have to fix quite some of that as a grandfather in the game there, but I would rather have a minimum sort of time before things go away as well. Yeah, actually, I would like to keep, I would like to guarantee keeping at least the last 24 hours with in case a certain idiot. That's a good idea. It should be pretty easy to add. Yeah. Although the next talk is... I was going to say the next talk, I think, is really, made me really relevant. It works very well, I must admit. I need to write it. I just need to make sure. Yeah. Don't be too too... I'm on the domain for a rewrite. You never started. No, the next... Actually, I talked to the next speaker about his Z-repl and it seems to be a much more formalized system around the snapshot and replication management piece, which I wasn't... It's brand new and he's kind of the only one using it right now, but I was very interested in actually picking that up and starting to use it as well because I think it may complement our set of tools really well. And now, the only thing I don't like about it is it adds in more complexity. This is already very, very simple and does accomplish the job, but if I can improve on it by... And the value is there in that improvement, then it's worth doing. Yeah, question. Yeah, so the question is how do we go about protecting against, say, a rogue system in or some kind of human error with an offline copy of the data? Right now, we just have the off-site, but it's not offline. That could very easily propagate some fat fingering across to the disaster recovery center. Yeah, there's the two cases. Something blows up that's fine, but malicious or accidents is a real concern as well. Yeah, I don't currently have an offline solution for that right now. I mean, one of the things we do... Yeah. I mean, for us, it is more important that we have the off-site had more up-to-date data though, too. Yeah. Yeah. Well, that's something we've thought about at the off-site center having a second Frenez box there that they do... The replica drops the data down, like, say, on a very specific schedule, so you have that protection of an additional copy there. Have you or anyone here looked at, say, two-factor authentication or even secure levels to prevent the pollution of snapshots? I haven't. And that's actually one of the things that I... Like, one of them... I don't know, not a problem, but like, one of my issues with ZFS is it's very easy and very quick to destroy. There's not even an ask you, like, if you care to destroy it. It's just... You've just destroyed it and it's very efficient. Thank you, Matt. But... I should almost... At least that on every machine, then. I could say for many times by the back, I almost always typo destroy. No one ever alias ZFSD story to destroy. But yeah, it is true. It's very easy to destroy things on ZFS. I have not looked, but once you destroy it, there's not like a, I can get it back type thing. It's gone. The snapshots, yeah, it's true. Which I have lots of snapshots, which is very nice. Yeah. So I'm going to say, how are you prepared to deal with, say, if a core telling you that a certain set of data must be destroyed from... Well, I'm saying it. I'm not at this point. It's very, very difficult to get rid of data in this setup. Luckily, you know, cross fingers. It literally happens better. Right? Yeah. So I don't even know, like, how we could go about that. You need to have a separate data set for every unit of data. Like, it took us a while to realize, but one data set for every single customer. Which is what we have. We do have one data set for every individual customer, because every customer has either a VM or a jail, and all the jails are, each of them have their own data sets. So a specific customer came to us and said, we need to have our data deleted, and yes, they can destroy all those data related to just their data. If they have a subset of their data, that's the only option to make them do VM. How can you get the data out? Right. That's probably exactly what I'd have to do, because it's just the data stays around. That, it has to protect you. You don't want it to. Yeah. Then you can put data sets. Yeah. I mean, the data sets are cheap, so use them, you know, all over the places, which is nice. Yeah. So, the way that I clean out unwanted data from backup snapshots is when I make a writable clone of the snapshot, then I eliminate what I don't want in there. You have to do that for every snapshot where the data is, and then you delete the original snapshot, and then rename, resulting snapshot back to the original name, then you actually have the right snapshot with the correct names. Yeah. So, you're saying it's possible to make a writable clone, delete the file, but you have to do this for every snapshot you would have. Could be, you know, we potentially have hundreds or thousands of snapshots. I would say that sounds like it's calling out screaming for automation. Yeah. That's actually one of my favorite features of ZFS is the.CFS directory. Yes. So, I don't know if you'll have an answer for this, but there's been very good news in the town. I've got some data where I have either contractual or regulatory requirements to put it on offline media, whether it be Word media or tape. Yes, I still do the tape world. How do you get ZFS data onto the tape efficiently nowadays? So, the ZFS data is a file, it's a serialized property. You can put it on tape and then receive it a year later. So, release ZFS and redirect to.CFS. Oh, pipe it to tar. Yeah. So, the question is how do you get ZFS data sets down to tape, and you can just stream it down to tape, ascend. Do you do the full under tape and you put it on the end of the issue? That's too easy. How would you like to set it apart? You know what tar stands for, right? Yes, that's true. ZFS has made our lives too easy. It's Michael. Are you using specifically LZ4, sorry, LZ4, and have you ever attempted deduplication that hasn't suited you? I've never attempted deduplication. We've had enough storage. That's not an issue. We are using LZ4. Nothing unusual? Nothing, I've never noticed. No. Is there something I should be aware of with LZ4? Yeah, there's something you're coming about. Well, I'm looking forward to the new stuff Alan is talking about. Yeah, that I'm actually quite excited about. So I'm happy to switch when the time comes. Yes. Have you given any issues? It sounds like a Zofi since most of the flat file data is. Have you given any cases where there's a process that's caching things heavily into memory and doesn't flush as often as it should? Where I mean to talk something like that. So the question is if we run into an issue where something is maybe not flushing down the disk as aggressively as we'd like with the application. It's not actually an issue with Zope. I mean now we're getting more into a Zope discussion specifically. Zope is an append-only database and it on start will automatically trim any corrupted transactions from the end of the database as it starts. So you can very reliably snapshot that file at any point in time. Even if the power outage happened or whatever it's very, very resilient to starting back up and basically rolling back whatever failed transactions might have been on the end of it. So for that we've just benefited because that's the nature of that application. If you have other databases where you want to ensure consistency like Postgres or MySQL I mean you should definitely have probably an additional cron jobs and in those cron jobs maybe to do some kind of a start backup, stop backup type thing. One more question and this is probably for Alan. Is there a way we can do this backup to force or if you have a ZIL to force it to flush? That I don't know. I think it's definitely a question for the right in front of you. What you're saying is the snapshot will be the same as the created one. Okay so the question is about the ZIL flushing on, but it happens automatically when you do a snapshot. And since you only send snapshots you have the whole data set. Always be consistent. I love that. Just to combine a couple of recent questions, slightly off topic data, but there's like an extra strategy for when they have backups on offline media like pay for CDs. And then they get a CDPR request telling them they need to remove all data related to the typical customer. I've only had a request like that once where we had customer data we shouldn't have had and it was in subversion. And so I had to filter it out of every subversion transaction to undo it, but I didn't have it on tape. We probably did, but we're using backill at the time. Yeah. Backill would have attended to that for you. So basically some of the data that I have to put onto archival media is government data. We have an unofficial opinion that as long as it is right only media, so word media, whether LTO or MDS, or anything like that, as long as it is not ultra bulk, the answer legally can be no, we can't do that. If you are able to offer the media, you are allowed to say no, we can't do that. So you choose your media carefully. This is yet to be tested. This is an unofficial opinion that we've been told verbally, but that's something you might explore through your legal team. Because almost all the legislation around the world allowing this does have some exemptions for where it is not technically possible to delete the data. And G8 is on optical media or something like that. Yeah. Couldn't you get into like a semantics thing if you have a check some stream? Absolutely. That's not changed. That helps. That'll be next year's presentation. You talk to your legal team, etc. But yeah, there are ways to do this that don't put yourself in the foot. And that's like the logical stream of like one database for each person. One database for every individual in Europe. Yeah. I don't know if your opinion law situation is, but this is under Canadian law and that's what we've been told. Whether it's mature or not, I am not a lawyer. Who in the room has gotten a notice to delete data? Okay. Coming. Well, not under GDPR, but under pre-project. Oh, actually, I'm sorry. We have season this, had season this since the years when the BBC project, the man with the most diverse workflows and stuff that people really want to experience. And do you have to go, you've gone all the way back through the backups of that? In that case, at the time, it was what backups, but... It was in CVSAR. Yeah. Globally distributed backups at the time. That's the official story. I'm sorry, the official line. So they do happen and yeah. I expected to get formal common works to figure out how GDPR is intended to be used. Yeah, this, I mean, I'm kind of worried about this because we do write applications that are portals where people log in and have a profile. And I don't know how to reliably say I've gotten rid of all your data. Like, there'll be these remnants. Or I can say, I don't know, with GDPR, do you have the time frame with which you have to guarantee it's all gone? I think the general rule is if you can make a case that what you're doing is very reasonable, they're more interested in the fact that you can do it and you're doing the right thing than specific doting guys and possibilities and specific timelines. Yeah. So you have a clause about, has to be commercially reasonable and you can't bankrupt yourself to leave the data. It's complicated not a lawyer. There's a lot of good write-ups on it. We do have a little case in talk to a lawyer. Discussion. Talk to a lawyer. Yeah. So talk to a lawyer is our answer. Almost makes you wonder if they didn't make this happen. Any other questions? Thank you all very much.
various backup systems and how we have backed up all of this is backups will be done when it is should make sure that backups are as simple to Amanda, we were able to do things like recover current packages we had installed. So we were on of Bacula, and Bacula is a very good product, but feeding to make sure that things are always and fast backups and making sure that they always supported some of these advanced features we were actually we were able to greatly simplify how we backup tasks, is it has to be reliable. We have to make sure that we can ensure that it runs and granular. We need to be able to get down to a able to restore a file quickly from that backup, of our infrastructure. Another key piece here is They know as little about the backup process than runs with little care and feeding. So when moving lines of configuration to manage that backup. And has to be backed up will have to have the backula combination of ZFS SNAP and ZXFOR to do the snapshot management and then the replication. Now snapshots, ZXFOR on a once a day process will servers. Now any server where we didn't relationship. Did you guys look at various at easily propagate some fat fingering across to the something you might explore through your legal