The BSD ISP
Stucchi, Massimiliano
Beck Peccoz, Amedeo
Running an ISP can be a tedious task of putting different pieces of boxed hardware together to make the network work, but can also be a fun and entertaining work of research on the right solution to accommodate your customer's needs. The market is full of vendors, big and small, ready to sell you a pre-packaged solution for your (supposed) needs, but what if you'd like to use BSD to serve your customers ? This talk will show how we are running a full ISP on solutions brought out using facilities and software easily built on top of BSD systems, and will delve into the challenges we have faced in the set up of the distributed architecture, with POPs in different european countries. Along with this, we are carrying out an analysis and comparison of costs and features between commercial and open source solutions, characterizing the decisions we made and the results we carried out. As an ISP, we not only offer access service, but we also deliver streaming services through a distributed CDN, also built on top of BSD. This will be a chance to delve into the different pieces of software used for encoding, distributing and streaming videos over the BSDs, and the technologies we used to interact with the underlying network. While being a non highly technical talk, the goal is to show the audience that using BSD in an ISP and content distributor environment is perfectly possible and will deliver the same quality of service of the packaged solutions, yet keeping your costs under control and allowing you a high degree of customization. This will be carried on showing - as already stated - a real world example of our project running solely with the power of BSD.
 Just one person. How many people? Two people. Three people. Y'all, you are sort of an ISP. Do you still work at ISC? Okay. How many of you do have experience with BGP? Okay. What kind of experience? Like running, configuring, public, no... Man, school. Sorry? School. Okay. Well, that is different from the real world. Well, agenda. So we have an intro background. We'll see how the background of what we've been doing over the last few years will go into details about some tools that we're using. And then we'll talk about the streaming services. So who are we? I am a RIPAA NCC trainer. I teach people about how to use IP addresses in the European region. I am a past founder of an ISP. So I created it from scratch. Everything configured from the first line of code from the first configuration line in our outer to... We were configuring bank networks. And I'm involved in networking some other projects. And Amadeo is the owner and operator right now of Bstream. He also founded the ISP 1997, the first fully automated ISP in Italy running on VSD. He's a coder. You can see him. If you go to this website, there's a live stream directly from Amadeo because he couldn't come over. So if you type it in your web browser, you should see a live stream from him. Wait a few seconds to type it. Sorry? Are you using Capitals? No. Well, he should be on one. I can see him. Can you see other folks in the wall? Yeah, yeah, it's coming. There he is. It's coming. There he is. Okay. I got that domain market.com, that stream.com. That stream. That's the original. I guess that's a totally different kind of thing. I don't know. You can do that. How do you watch the live stream? Well, later. So what is this about? The idea is to open up discussion for people to use open source software to drive their business. In the networking world, when you start mentioning, oh, I run a software router and Mike knows about that. People look at you and they say, ah, sometimes you have even problems bringing your own hardware into IXPs because they don't want that. They don't know what it is and it's a problem. I found this to be a problem in my previous job. So what I'm trying to do, what I generally try to do is to get people to start using software solutions to overcome problems. And we'll see why then. But so I'd like you to participate. So if you have questions, if you want to give me insights from your, or if you want to just raise your hand, talk, participate, and you can also set up an hashtag on Twitter. And there you go. So why are we doing this? Because I've seen it works. And because we're going to show our experience with it, well, some of it. And ultimately, because it's definitely cheaper than other solutions. Definitely cheaper, depending on how you look at it. If you look at it from the perspective, from the real money, on the money side, yes, if you look on the operational side, you have to train people and we'll see that later. What is the background? Streaming is a streaming company. It is based on FreeBSD. And for streaming, especially live, you need connectivity. So we were, me and Amadeo have been friends for many years. We were discussing about this after I left my previous company. And he was telling me that he was having problems finding good connectivity in Europe for some of the streams he was supposed to do. And then I came up with an idea. Let's try to become an exercise speed. We won't be discussing about the paperwork here. There's a lot of paperwork involved in Europe to become an operator, to get licenses. It's different from getting a license from the FCC in the US, but it's still hard. And it takes a lot of effort. But it can become a major selling point when you're trying to go to one of your customers, selling them a streaming service, and you come up with a complete solution that will bring you the connection service, will bring you the access, and will also give you the stream. I don't know how you use it, but I'm doing what I do with this. You don't deal with... Okay, but how do you deal with this when you go to your customers, when you're asked to run a live streaming or... Sorry? So you don't do that? Okay, why not? It's not in your business plans? I'm not in the business plans. Okay, so you don't deal with it? Okay, perfect. Sorry for the question. What were the goals? Provide access along content. Keep the costs low. So we said, okay, let's try it. Let's try it if it works, and then if it works, we'll get bigger hardware. We'll get more, and we wanted to use technology which could grow with, because the problem, and we will see later, is that some of the technology you can buy nowadays will be obsolete in a very short period of time. We wanted to mix up content and eyeballs. When you do streaming and you're sending lots of bits across the networks, you're sending them out of your network. But when you start doing access, you will begin to have traffic coming into your network, and so you will have a balanced... The goal was to get a balanced ratio between outgoing and incoming traffic, so that when you go to PIRIT on IXP, you will get more attention from the other networks, because they won't have to deal with... Well, how can I put this? They will more likely PIR with you than when you're just a content network, and you will have also eyeballs for their contents, so if you have a balanced ratio of traffic, it will be easier for them to deal with you. How would you achieve that balanced ratio of traffic if you have to start small and you have to start somewhere? So really you're starting with content delivery now, right? And then as your clients grow, you're saying then, is it your staff to plan PIRIT arrangements with other upstream... Okay, no, the problem is when you do this, you act with hot potato routing, so what you will do is you will hand out your bits, your content bits to the closest point of PIR with any other network, and then they'll have to deal with bringing your bits into their network and bringing them to their users, and that's a major deal. So you're sending more than receiving, and this is a cost for them, but when you have a balanced ratio, you will see you as a point where they can send you bits as well. And so generally, let's say you get to a new internet exchange and you send out an email, hey, I want to PIR with you guys. What generally the answers you will get are, smaller networks will say, yes, okay, we'll PIR with you. No problem, we'll set it up, but the amount of traffic will be very low. Bigger networks tend to tell you, hey, yes, we could PIR with you if you meet these requirements. And the requirements generally are, I don't know, one gigabit of traffic, symmetrical, so not just in one direction, PIR over multiple internet exchanges in different regions of the world. So sometimes you have to be big to PIR with them. Sometimes you can just try and tell them, hey, I have, I don't know, three gigabits of traffic, but in just one region, would you PIR with me? Because I see you have eyeballs or you have content. So generally it helps if you, well, we'll see this later, if you generate graphs based on net flow output and look at the destination and origination ASNs for the traffic you generate or receive, and you show them, hey, I have two gigabits of traffic coming from your network into mine, but I don't have any traffic coming out of my network into yours, would you like to PIR with me at this internet exchange? So other than passing through forensics, we will just exchange traffic this way. So it's a matter of giving them a chance to see the value in managing a PIR session with you. Because remember, if you're a big network, you will have thousands of sessions to keep up unless you find internet exchanges with good and valuable route servers, which is a different kind of story. Do you guys know what a route server is at an internet exchange? I don't see many ads shaking. Okay, so, well, Mike knows. If you go to an internet exchange instead of PIRing with all the different entities, you will PIR with just one machine, one session, and you'll get all the routes from all the other participants, and then you will send traffic directly to their interfaces. So instead of having, say, 200 sessions, you will just have one, but you will receive routes for all the other participants, which makes life easier. For example, Google only PIRs at internet exchanges with a route server, and otherwise they would have to manage thousands and thousands of sessions all the time. And you turn the drive to the internet. I don't know about that. I don't know the internals, but we're not here to discuss the internals. That was just an example. So, does this answer your question? Is there any other questions so far? No. So the first steps are define a routing policy, define Xcon, and cross connects for XDSL services and related protocols and standards, because there are a bunch of them, define new goals for the CDI content delivery infrastructure, and shop for vendors and or software. So, routing, how do you think about doing it? BGP, of course. Maybe OSPF as IGP, although we ended up with a different solution with RIP for the moment, and as you can see from my t-shirt, I wanted to do IPv6 from the beginning. So the idea was to begin with IPv6 and then do IPv4 on top of that. What are the routing challenges nowadays? The routing table is continuously growing. It's almost touching 500,000 routes. We are 556,000 at the end of this week. We at the beginning was 430,000. You have to be IPv6 enabled if you want to be future proof. And if you want to be IPv6 enabled, well, there will be more routes to come. And it will be fun. I mean, I'm really wondering what will happen when people will start announcing Slash48 separately, because this will mean that any provider on the world might be able to announce up to 65,000 different routes. So I imagine you multiply it by the number of ASN numbers and the number of networks that are out there. So the problem is with vendors, many of them, they have T-cams, so the memory on the routers, is so far limited on the low-end models to 1 million IPv4 routes. So if you have more than... If the routing table goes to more than 500,000 entries and you have more than one transit, you won't be able to store the whole routing table on your router. So the problem is it will start to save them memory and it will be slower and slower and slower in loop backups. And add to this the fact that you also want to have IPv6 routes. So you will also have even more and then your machine will just slow down. Unless you want to cash out money for higher-end equipment, which means, of course, getting more routes and getting more money out of your pockets. So how do you solve this problem? You can use Quagga, OpenBGPD, BERT, and ExaBGP. Quagga is the closer to the Cisco implementation. Quagga has a DemonRunning and then you have an interface which is very similar to the Cisco interface. So if you have somebody who is familiar with that, it's very easy to move them from a pure Cisco router to a Quagga-based router. But it has way many more bugs than I would have liked to know and see or to hear about. You constantly hear about problems in the Quagga base code. Even though they're fixing them, they're still out there. So I wouldn't recommend it. So OpenBGPD, instead, is native on OpenBSD. And remember we said the shop was freeBSD based at the time. And at the time the port was lagging behind inversions on freeBSD. And the here present mic, he's denying it now, but I asked him about, he was using OpenBSD at the time. OpenBGPD knows OpenBSD. And he reported me that in his network he saw string things happening with OpenBSD. So even though I was already using it in MENA, it's an internet exchange we have in Italy that I founded along with four other people, I didn't feel comfortable using it because of this records, because of some tests that I made internally, which didn't quite give me the expected results. OpenBGPD is solid. OpenBGPD is solid, yes. OpenBSD is less solid and sometimes it will lose state of the interface and start, you would unplug an interface and not see it going down in your internal IGP. BERT was mature enough at the time, promising enough to be able to handle MPLS and multiple tables. We were already using it at MENA, we were using both BERT and OpenBGPD. It also supports OSPF and GRIP, so this is why we ended up using GRIP. I don't know if one of my guys said this is okay, this can't put that good on the desktop. Might be, yes. But so far I haven't seen any problems. You haven't laughed, so I mean... No, well, even with... Well, yesterday was two days ago, we had some major flapping on the Internet, like six million route changes in a few hours. Yeah, I didn't even notice. So just for you to know, there were reports in the last few days of instability on the Internet, so you would get lots and lots and lots and lots of updates. So withdraw routes, add routes, withdraw routes, add routes, and this makes your CPU usage go weary when this happens and six million changes in a few hours is a lot. So we end up using BERT. ExabGP instead is not really a BGPDmon, it's just a toolkit. The idea was to use it for the CDN, so to do anycast and also to play with communities and backhauling, blackhauling. So what ExabGP does is it runs as a very, very, very tiny demon where you can originate announcements and you can accept announcements and do whatever you want with them, you can script it. So we ended up writing a solution where we are doing blackhauling. So this means that we inject a route into our IGP where ExabGP redistributes it into our border routers and with a specific community telling our border routers to null route this specific IP so that if we are getting, I don't know, scanned from one of those IPs, we just blackhaul that and we don't receive any traffic anymore and after some time we just bring it back into, we cancel this route specifically. So ExabGP works fairly well, you can also use it to test your BGP implementations or to test your routers because you can inject even the full routing table and have it perpetrated into your system. So let's go on with XDSL aggregation. The goal was to do it in different countries, we're actually doing it in three. One started it even across countries so every major incumbent you can find will do L2TP so L2TP with PPPoE is a termination. I guess they also do it in Canada, right? So this was quite easy, there's MPD in FribuC, it's easy, simple and stable, never had any problem with it. A configuration is very, very simple and if you need any help, Alexander is very, very, very, very, is available. So you can just send, shoot him an email if you have any problem with the code or anything. We actually had problems with some parts of the code, we emailed him, came back with an answer in less than 24 hours and then we worked together to fix the problem and we came up with a good solution. MPD can be interfaced easily with Radius so if you have users in Radius, you can use it. You can have it look up in Radius, we also have a modification to MPD, we modified a little bit of the code to do some, to tunnel again L2TP to continue, to carry on the tunnel towards some of our customers because we have, at one point we started having resellers, so we, sorry? No, it doesn't do tunnel switch. So we got the call in from, open up the L2TP tunnel, we do, we look into the PPPOE session, get the username and domain based on the domain, we forward it to another, well, but we do it dynamically looking at announcements with CVBGP, what we have in the PPPOE table from this customer before that to see a set of other service they wanted to do it. They said we have it already working with other implementations in their French, so I won't get into any more details. So other stuff you might need, these are tips from somebody who has run an ISP for some time. So you want an IP address management or config manager, a looking glass, a NetFlow, anyone? Do you do a NetFlow? No. Sorry? We didn't have an X-ray. You want some recommend management software and diagnosis tools? Sure. Yeah, that's into, that's actually, that's what recommend management means. I don't understand it, it's incredible to see how many times it has people with RACS servers and only a fellow server with login with the justice team. Yeah, well the idea is to label everything. People who have bigger operations might already know all of this, but recommend management is one side, what I, how I called this. There are different softwares, there is, well, IPAM and config manager, I ended up writing my own internal solution, even for RAC management and cable management. So, yeah, I think that's the next section, Simon put some slides of some disasters, some interesting examples of how that works. I actually had a ring label that I don't know where everything is, I don't have to get it, so we'll explore and find everything. Yeah, well, the thing is when you're a bigger shop, you already have something in place for this, but since we were actually, It's something that's found in the dump, it's far over, it's not your black community. Yeah, exactly, exactly. So what we, the problem here was that we already had some servers scattered all over and we started adding more, more equipment to do the BGP part and the DSL aggregation. So we started needing something like this and when you, when you start adding transit, so two transits per, per pop in different countries, then you want to start having something, some way to know where exactly where things are. Yeah, of course it happens that you call the colo and have the wrong machine rebooted, but that still happens, but it should not. Sorry? It does, it will happen. I recommend subnet's manager because it does both IPv4 and IPv6, it's very easy, it's written PHP, sorry, I didn't write it. And if you live in the error in region, it can do, it can already send the, the swips directly to error in to register your assignments, your assignments to your customers. I'm writing to integrate the ripe way of doing things in the, in the software, but not yet. Do you guys know what the looking glass is? The, looking glass is, the purpose of looking glass is to be, to look into what's happening in your routers or outside of your routers. If you are, LG is the de facto standard but needs remote console access, so that means that it's based on Cisco, Cisco gear. So it has to connect to the, to the console on the Cisco gear and run comments. And if you're not running Cisco gear, you can't run it. So GI XLG is a reasonable solution, it's based on XBGP, so that means that you might want to have XBGP running on the same machine as your, as your other, it's based on GX, it's based on XBGP and uses MySQL as back end, so you can do, you can have XBGP put all the routes in, in MySQL and then look them up. NetFlow, PF flow D, I would have liked to see Henning do it, because he committed it a few years ago and then never touched it again. I don't know if it works, I never tried it, so actually for NetFlow there's not much going on, on, on the PSDs. So if you have any hints or ideas, I would be glad to hear about them. And, sorry? Five years ago, I took the one in NetFlow. No. So, various others are, for example, rack management. My advice is to use RackMonkey if you don't want to rack your own, because it's great. It lacks actually the ability to, to track cabling, but you can easily, you can easily manage it via some of the, the notes, the notes system it has or you can manage, you can manage it. It's still in RackMonkey. It's not as, it's not as good as it would do if it was written for that purpose, but still you can do it. Smoking, because when a customer calls in and says, oh, my connection is not working, you want to tell them I see that it is working or not. MRTG cacti, depending on size or your choice, whatever system you want. So now, I think it's Amadeo's time to go into details about the content delivery network. Let's see if that works. Can you hear him? Yes. Can you guys see him or want me to put him on the, on the Vix screen? Hello? Can everybody, can you hear me? Yeah, they can hear you. Can you hear me? Yes, we can hear you. Just go on. Okay. And I think we have a problem with the audio, I don't know, so if you need to tell me anything, use the keyboard. Tell me here and what you think. And I think we have a problem with the audio, I don't know, so if you need to tell me anything, please use the keyboard. Okay. Okay. So, while we had this ISP in place with multiple locations, we also thought of using it to... So, we reflected. So we've done a VSD-based... While we had this ISP in place with these multiple locations, it's right on the SCAOS screen too. Yeah, we've done a VSD-based... Hold on, hold on. In this context, we know it can't. So we do everything, we do all the information, we're just reading, and then the VSD-based... Sorry?...is made... Yeah, we tried last night. Now, just a quick overview of what a CDN is. A CDN is made essentially... Okay. Okay, just a second. Okay, now it should be... Okay, with the audio. Now, a CDN is made out of basically two pieces, a CDI, which is the infrastructure, what actually delivers the beat. And then a Condom Management System that allows you to decide what to deliver and how to deliver, and to provide value-added services to the end user. So, what we have, the first thing in network design, if you want to do a CDI today, you can ask yourself, shall I make it or buy it? When we started all this back in 2006, to buy it was not really an option, because there were very, very few players, and they were sort of extremely expensive, and with little, if any, flexibility. So, it was not really an option to buy the Condom Delivery infrastructure from someone else. Today it is. You might have a look at Cloudflare, for example. Also, there is some other services from Amazon, at least for the basic delivery service. I think if you want to do a CDN as a business, so if your core business is CDN, you need to run your own CDI. Otherwise, you are just a sort of reseller for someone else's services. So, you have different options if you want to make your own CDI. Go for a single server, now this is of course an option, but it is not a real one, because the real reason to have CDIs is to reduce the number of hopes and to deliver the bits as close as possible to the end users. Also, you might consider a single delivery center, which is more or less the same thing. So, don't go for it. What we do, we have multiple delivery centers, and that's what we will talk about. So, okay. The first thing you need to find a way, a technical way to make sure that the end users can reach the closest available version of the content that they are looking for. And so, you have different options. You can use ITER and IP routing, so that using routing, BGP basically, you can direct the user to the closest available servers, or you can use DNS. We've been using both, and definitely the routing option is the best one. Everything again, based on a BSD solution. The second element that you need, you're doing really a network over the network, because you need to be able to reach and to deliver similar content across your network, your servers. So, you can ITER do it, roll to say so, so you go on the internet, or maybe for some services, you might need an open VPN or an IPsec solution. Beware of layer 2 or level 3 services that you might buy around, they call it VPN as well. I do not recommend layer 3 services, because they don't add anything. They might also be slower or less reliable than a standard internet connection. We use open VPN to send, for example, accounting information or content localization. You know the location. You need these because you need to be able to have local information. I mean that, and also local content. Every distribution center needs to know everything about all the content that needs to be distributed, where it is, how to locate it, and also need to have the information about accounting, for example. If you want to disable a certain user or certain content, the information from our point of view needs to be locally present and replicated and distributed in every distribution center. You don't want to keep it centralized. Because of, of course, if you have a failure somewhere, you need to be able to continue the service. Now, for content delivery infrastructure, you need software. The hardest part is to find a streaming solution, a video and audio streaming solution. Today, you basically have three options. You can read them here. Red 5, Wozze and Adobe Flash, Flash Media Server. They're all Java based, so you can run them on any standard BSD platform with Java enabled. Red 5 is open source and free. On the other hand, it only provides delivery to RTMP, which means Flash devices. The other two are more or less the same thing. Adobe, possibly a little more expensive, and Wozze is the one we use because it's not that expensive, and it offers a higher number of delivery options for recovery. You can stream to mobile devices and like. HTTP, if you need to deliver HTTP that's very basic, you use your preferred web server. Maybe you need to write your own, I mean, both cases for streaming and HTTP, you need to write your own accounting solution. When you run a CDN, basically you sell bits, so you need to make sure you count how many bits go out to be your clients. And also, you need to write plugins for the streaming services or application or whatever you call them, to make sure you're able to locate the content, which server and which distribution center it is in. And then to provide a number of value added servers, we'll come back later about the value added services that you can do. Coming to the content management system, many people say, okay, that's the easy part. If you've done the content-ravelling infrastructure, content management is the easy part. I don't agree completely and it says that it is easy if you want to make just the basic things. It can become a little bit challenging if you want to deliver especially for the value added services that we'll see in a moment. Anyhow, you need to choose your web development platform because CMS is usually based on the web. You might be willing to write an API via web services or whatever other solution you want to use to integrate with your content-delivery infrastructure. You need to keep an eye on accounting so that accounting information is available via the content management system. And then you need to handle publishing. When we talk about video streaming, usually people have the idea of a sort of YouTube where you automatically, whenever you upload something, it's automatically you can view it. Everybody can view it. You just copy and paste. You are out to your friend and they can see the video. Now, we decided to split a little bit the publishing from the uploading. So you're responsible to publish the content on your own. This is a sort of big difference because we focus mainly on B2C market. We don't go directly to the end user. So you want, for example, to be able to have a wide label services that you want to rebrand. Or maybe you want to show that your company is now a relationship with our streaming platform. So the publishing is left aside. The value-added services, here I mentioned a few, you can do multi-conference with streaming. Streaming can be only directional as it is now, or it can be also bidirectional. Now, if you multiply with many users, you can do multi-conference. It's a little bit tricky. It's not that easy, but still it's possible to do it. For example, also you can do content tagging. We've been using content tagging with a client or farce, and they have a lot of video, so they need to be able to recognize them somehow. As a scontrol, for example, let's say you want to allow only invited people to see your video. And maybe content encryption, because maybe you want not your delivery sensible videos, for example, meetings or things like that, and you want to use encryption. Now, one of the toughest part, and with this I'm about to complete this short speech about streaming, how to do content preparation. Now, content preparation is when you have videos, or I mean generally speaking when you have content, but the hard part is to deal with video and audio. So, the minimum toolset that you need to use is GIPPAC, FFMPEG, MPLayer and Gstreamer, which is not mentioned here. You want to be an expert with these tools, because you always need to convert, transform. There is no real defined standards in the video streaming industry, and many new standards pop up continuously. So, you need to sort of convert and make sure you can transcode or adapt the video for video streaming. Now, if you use FreeBSD, you might be willing to have a look at user ports multimedia and user ports audio. There is a lot of stuff that is very useful. And then, if you want to do cool stuff with videos, you should really be using user ports multimedia MLT. This allows you to write scripts that can add effects to a video, merge videos together, show things, popping up and down in video, change the audio, make effects and the like. So, it's really powerful tool. And the last slide is about the script that is running this conference. It's running on BSD, uses standard stuff. We've been using a camera connected to a laptop with a firework cable. I'm making a copy of it, so you can see the T-Common. I'm making a copy of the video, so if I want to work and use it afterward. And then, I'm encoding live with FFMPEG directly into our ingest point. And that's it. Now, should you have any questions, please ask them to Massimiliano and Massimiliano. Any questions? How do you guys handle the broadcasting? I'm hearing that you use this same method. Wait, somebody else. How? You need to define the edge, because the edge of the internet is, as far as I understand, not a unique place. It depends on how you define the edge. The edge is where we have a content delivery, content distribution center. That's what we call the edge. Now, we don't have a centralized structure. So, for example, when you upload a video on the... Yes, okay. When you upload a video via the content management system, it is automatically distributed to all the content distribution centers. Now, what happens if the content is not requested for a period of time, it might get deleted on some content distribution centers. And then, we know how to locate it. And if it is requested again, we copy it again, we send it again to the content distribution system. And how do we handle this? This is something we had to write and we had to do on our own. It's based on FuseFS. We're going with some final considerations which are mostly related to the networking part, a little less about the streaming part. So, beware. While it might be easy to set up, you have to maintain this kind of environment, which means you need to have knowledgeable people. One of the challenges that I found at my previous job was that I had some employees who were networking guys. And I told them, okay, let's implement this software routing system. And they were scared about it because they didn't know where to start from. Because they had never seen a Unix console or whatever. And so, that was a major challenge in trying to implement this. And this is also one of the reasons why I had to... Well, this is one of the reasons why at the end I decided just to leave. Yeah, I'll leave and sell my shares of the company. So, many networkers know Cisco and Juniper and maybe a little bit of MicroTik, but they don't know Unix. They don't have the knowledge or the need to get to know that. Because if you tell them to go learn this, they will just switch to another job. And this is something you want to keep in mind. And VGP is a threat. Well, you could be a threat to the internet. So, you have to know what you're doing when dealing with VGP. So, who designed it? Whoever designed it thought that you have to be knowledgeable to run it. You can do a great deal of damage to the whole internet when you're running VGP. So, take for example the YouTube against Pakistan Telecom incident. Somebody in Pakistan started to announce the YouTube prefixes. And so, all the traffic directed to YouTube went to Pakistan instead of going to YouTube directly. And there are newbies playing around with MicroTik. Instead of doing, they created an ISPath of 256 entries. And they tickled the bug in all the other Cisco routers all over the world. And sessions all over the world were flopping around. And it took a while to figure that out for the whole internet. So, you know, you can't just email going back and forth to between people. Sorry? They were newbies in MicroTik. And didn't want to run, didn't want to read the manual. So, and so on. There are incidents like these pretty much every day. Some have a smaller influence on the internet, some have a bigger one. And yes, we showed it as possible. Now it's up to you. And one of the things I'd like to highlight is if you run, if you have a network, if you have an ISP, you want one of these probes to be in your network. Because there, this is a project from the RIPE NCC to monitor the internet. So, if you want one, just let me know. And I can ship you one from my office. So, you can install one of these probes at home in your office network. And you can monitor the, you can use them to monitor the whole internet from there. Or you can, you can use, you can use a, you can, you can play with one of these probes and get points using them to run your own test measurements against your own network. So, if you want to say, I have 500 probes all over the world, ping one of your IP addresses to see the reachability. You can do that from the, from the Atlas website. Or you can, you can run any, any other kind of measurement. So, it's, for now you can only do ping trace route and DNS lookup. But it's going to become a, it's a toolbox to run your own measurements on. World effort worldwide. Do you have any other questions? No questions? We're either. I just, looking at some of the technologies that you're using. And just from what I've read. Yeah. And it seems like a lot of the technologies you're using are some, not most, are the same that are integrated into ISP today, and like, correct, with the exception of somebody obviously, the content of the pre-infrastructure component. You mean for the, for the streaming part? Well, ISP config is more for, it's more suitable for hosting, hosting sites. So that, I didn't get into that detail because I was mostly focusing on the access part. But yes, some of, most of that is, well, I didn't get into the web server side of the, of the operations, but web server or email or whatever else side of the operations. But yes. What do you do with the accounting, the DNS? Yes, well, well, DNS, when you do access you want to just generally manage only reverse DNS for your customers. And again, I ended up writing my, my own software. It was actually integrated into the IP address management software where customers could just log in and see their IP addresses and set up the reverse DNS. It's trivial. Very, very, very trivial. So, yes, I didn't get into that detail because I was focusing more on the routing part. But yes, it's, it's mostly standard software. Yeah. So, yes, I'm, well, BGP is, why do they have that? So, yes. So, for the rest is pretty much common. Any other question? Do you build a time on the transit? Yeah. No, not the transit. A transit we do on antico-person-tile to, well, we have some co-location customers and we do on antico-person-tile on them. How do you build a time on that? MRTG. Sorry? Yeah. We're not very strict because actually, as, again, we go back to the content versus eyeballs, content and eyeballs. If you, if you keep a good balance between outgoing and incoming bandwidth, you don't want to care about what your customers are really doing. I mean, well, you still have to keep an eye on that, but you're not so strict. Unless you really want to charge them based on each and every megabit that they're doing, there's no reason to do that. So far, we do it just look at the graphs at the end of the month and that's it. Yeah. Yeah. Yeah. Yeah. Same thing. Unless, unless they're, they, they bought like 10 megabits of transit and then they do 20 or 25, well, then yes. They generally send them an email telling them, hey, this month you did this person pays more than you should be allowed. Are you guys comfortable with the 95th person tell how it works? No. Okay. So basically you run MRTG and you, and you, you get a sample of the traffic every five minutes. So you, at that point, you know how much traffic has been passing by on that, on that interface. And at the end of the month, you discard the highest 5% of those, those readings and then you get the average out of that. So you see that on, on average, your customer has done 20 megabits and you discard the 5% higher readings. So that means that he can do for 36 hours a month, he can do say if he bought 10 megabits of transit over 100 megabits port, up in port, they can do 100 megabits for 36 hours a month. Which means that if they have spikes or if they, if they want to do, I don't know, advertisement campaigns, they can still do them and not be built for them. And while you, while you might have big, big spikes on your network, you can still be sure that if they do more, if they use more traffic, then they will get built, or you can build it. Sorry? Exactly. Easy to put into this, you get the old, the old, the old 30 megabits. So. Quite a long, this may already be 30 hours. But what, what generally happens is that, I know some, some of the customers will pay 20 megabits and then the 95th person to be able to do 21 on average. You don't want to start billing them for 21. You just play the good, the good part and send them an email and say, hey, you know, you've been doing 21 megabits and then you might think about getting 25 of transit instead of just 20. And by the happens rarely because, well, either you are proactive or when you have both content and, and access on your, on the same network, you can, if you start having a lot of access, you can start actually giving content bandwidth away for free. You can start playing with it like using it as a commodity to encourage customers to stay with you. Sorry? Yeah, some costs because you're, you're always paid for, for bandwidth in both directions. So you will, if you're, if you're having access and you have 30 megabits of traffic for your access network, you will have 30 megabits getting to your network. And then you will still not, you might not still be sending too many bits out of your network. So if you, if you get co-location customers and you, and they will start sending bits out of your network, that's bandwidth you're already paying for. And then you can start playing with it from a sales perspective, sounding it for very cheap. Well, there's a case here in Ontario, Canada, there's a very large business, a very large business model, I mean, if anyone supported around code, many did second-hand code pump from 2007, Thank you.
 Just one person. How many people?  Two people. Three people.  Y'all, you are sort of an ISP.  Do you still work at ISC?  Okay. How many of you do have experience with BGP?  Okay. What kind of experience?  Like running, configuring, public, no...  Man, school.  Sorry?  School.  Okay. Well, that is different from the real world.  Well, agenda.  So we have an intro background.  We'll see how the background of what we've been doing over the last few years  will go into details about some tools that we're using.  And then we'll talk about the streaming services.  So who are we?  I am a RIPAA NCC trainer.  I teach people about how to use IP addresses in the European region.  I am a past founder of an ISP.  So I created it from scratch.  Everything configured from the first line of code  from the first configuration line in our outer to...  We were configuring bank networks.  And I'm involved in networking some other projects.  And Amadeo is the owner and operator right now of Bstream.  He also founded the ISP 1997,  the first fully automated ISP in Italy running on VSD.  He's a coder.  You can see him.  If you go to this website, there's a live stream directly from Amadeo  because he couldn't come over.  So if you type it in your web browser,  you should see a live stream from him.  Wait a few seconds to type it.  Sorry?  Are you using Capitals?  No.  Well, he should be on one.  I can see him.  Can you see other folks in the wall?  Yeah, yeah, it's coming.  There he is.  It's coming.  There he is.  Okay.  I got that domain market.com, that stream.com.  That stream.  That's the original.  I guess that's a totally different kind of thing.  I don't know.  You can do that.  How do you watch the live stream?  Well, later.  So what is this about?  The idea is to open up discussion for people to use open source software  to drive their business.  In the networking world, when you start mentioning,  oh, I run a software router and Mike knows about that.  People look at you and they say,  ah, sometimes you have even problems bringing your own hardware  into IXPs because they don't want that.  They don't know what it is and it's a problem.  I found this to be a problem in my previous job.  So what I'm trying to do,  what I generally try to do is to get people to start using  software solutions to overcome problems.  And we'll see why then.  But so I'd like you to participate.  So if you have questions, if you want to give me insights from your,  or if you want to just raise your hand, talk, participate,  and you can also set up an hashtag on Twitter.  And there you go.  So why are we doing this?  Because I've seen it works.  And because we're going to show our experience with it,  well, some of it.  And ultimately, because it's definitely cheaper than other solutions.  Definitely cheaper, depending on how you look at it.  If you look at it from the perspective, from the real money,  on the money side, yes, if you look on the operational side,  you have to train people and we'll see that later.  What is the background?  Streaming is a streaming company.  It is based on FreeBSD.  And for streaming, especially live, you need connectivity.  So we were, me and Amadeo have been friends for many years.  We were discussing about this after I left my previous company.  And he was telling me that he was having problems finding good connectivity  in Europe for some of the streams he was supposed to do.  And then I came up with an idea.  Let's try to become an exercise speed.  We won't be discussing about the paperwork here.  There's a lot of paperwork involved in Europe  to become an operator, to get licenses.  It's different from getting a license from the FCC in the US,  but it's still hard.  And it takes a lot of effort.  But it can become a major selling point  when you're trying to go to one of your customers,  selling them a streaming service,  and you come up with a complete solution  that will bring you the connection service,  will bring you the access, and will also give you the stream.  I don't know how you use it, but I'm doing what I do with this.  You don't deal with...  Okay, but how do you deal with this when you go to your customers,  when you're asked to run a live streaming or...  Sorry?  So you don't do that?  Okay, why not? It's not in your business plans?  I'm not in the business plans.  Okay, so you don't deal with it?  Okay, perfect. Sorry for the question.  What were the goals?  Provide access along content.  Keep the costs low.  So we said, okay, let's try it.  Let's try it if it works, and then if it works,  we'll get bigger hardware.  We'll get more, and we wanted to use technology  which could grow with, because the problem,  and we will see later, is that some of the technology  you can buy nowadays will be obsolete in a very short period of time.  We wanted to mix up content and eyeballs.  When you do streaming and you're sending lots of bits across the networks,  you're sending them out of your network.  But when you start doing access,  you will begin to have traffic coming into your network,  and so you will have a balanced...  The goal was to get a balanced ratio  between outgoing and incoming traffic,  so that when you go to PIRIT on IXP,  you will get more attention from the other networks,  because they won't have to deal with...  Well, how can I put this?  They will more likely PIR with you  than when you're just a content network,  and you will have also eyeballs for their contents,  so if you have a balanced ratio of traffic,  it will be easier for them to deal with you.  How would you achieve that balanced ratio of traffic  if you have to start small and you have to start somewhere?  So really you're starting with content delivery now, right?  And then as your clients grow, you're saying then,  is it your staff to plan PIRIT arrangements with other upstream...  Okay, no, the problem is when you do this,  you act with hot potato routing,  so what you will do is you will hand out your bits,  your content bits to the closest point of PIR with any other network,  and then they'll have to deal with bringing your bits into their network  and bringing them to their users, and that's a major deal.  So you're sending more than receiving,  and this is a cost for them,  but when you have a balanced ratio,  you will see you as a point where they can send you bits as well.  And so generally, let's say you get to a new internet exchange  and you send out an email, hey, I want to PIR with you guys.  What generally the answers you will get are,  smaller networks will say, yes, okay, we'll PIR with you.  No problem, we'll set it up,  but the amount of traffic will be very low.  Bigger networks tend to tell you, hey, yes, we could PIR with you  if you meet these requirements.  And the requirements generally are, I don't know,  one gigabit of traffic, symmetrical, so not just in one direction,  PIR over multiple internet exchanges in different regions of the world.  So sometimes you have to be big to PIR with them.  Sometimes you can just try and tell them, hey, I have, I don't know,  three gigabits of traffic, but in just one region, would you PIR with me?  Because I see you have eyeballs or you have content.  So generally it helps if you, well, we'll see this later,  if you generate graphs based on net flow output  and look at the destination and origination ASNs  for the traffic you generate or receive, and you show them, hey,  I have two gigabits of traffic coming from your network into mine,  but I don't have any traffic coming out of my network into yours,  would you like to PIR with me at this internet exchange?  So other than passing through forensics, we will just exchange traffic this way.  So it's a matter of giving them a chance to see the value  in managing a PIR session with you.  Because remember, if you're a big network, you will have thousands of sessions to keep up  unless you find internet exchanges with good and valuable route servers,  which is a different kind of story.  Do you guys know what a route server is at an internet exchange?  I don't see many ads shaking.  Okay, so, well, Mike knows.  If you go to an internet exchange instead of PIRing with all the different entities,  you will PIR with just one machine, one session,  and you'll get all the routes from all the other participants,  and then you will send traffic directly to their interfaces.  So instead of having, say, 200 sessions, you will just have one,  but you will receive routes for all the other participants,  which makes life easier.  For example, Google only PIRs at internet exchanges with a route server,  and otherwise they would have to manage thousands and thousands of sessions all the time.  And you turn the drive to the internet.  I don't know about that. I don't know the internals, but we're not here to discuss the internals.  That was just an example.  So, does this answer your question?  Is there any other questions so far?  No.  So the first steps are define a routing policy, define Xcon,  and cross connects for XDSL services and related protocols and standards,  because there are a bunch of them, define new goals for the CDI content delivery infrastructure,  and shop for vendors and or software.  So, routing, how do you think about doing it?  BGP, of course.  Maybe OSPF as IGP, although we ended up with a different solution with RIP for the moment,  and as you can see from my t-shirt, I wanted to do IPv6 from the beginning.  So the idea was to begin with IPv6 and then do IPv4 on top of that.  What are the routing challenges nowadays?  The routing table is continuously growing.  It's almost touching 500,000 routes.  We are 556,000 at the end of this week.  We at the beginning was 430,000.  You have to be IPv6 enabled if you want to be future proof.  And if you want to be IPv6 enabled, well, there will be more routes to come.  And it will be fun.  I mean, I'm really wondering what will happen when people will start announcing Slash48 separately,  because this will mean that any provider on the world might be able to announce up to 65,000 different routes.  So I imagine you multiply it by the number of ASN numbers and the number of networks that are out there.  So the problem is with vendors, many of them, they have T-cams, so the memory on the routers,  is so far limited on the low-end models to 1 million IPv4 routes.  So if you have more than...  If the routing table goes to more than 500,000 entries and you have more than one transit,  you won't be able to store the whole routing table on your router.  So the problem is it will start to save them memory and it will be slower and slower and slower in loop backups.  And add to this the fact that you also want to have IPv6 routes.  So you will also have even more and then your machine will just slow down.  Unless you want to cash out money for higher-end equipment, which means, of course, getting more routes and getting more money out of your pockets.  So how do you solve this problem?  You can use Quagga, OpenBGPD, BERT, and ExaBGP.  Quagga is the closer to the Cisco implementation.  Quagga has a DemonRunning and then you have an interface which is very similar to the Cisco interface.  So if you have somebody who is familiar with that, it's very easy to move them from a pure Cisco router to a Quagga-based router.  But it has way many more bugs than I would have liked to know and see or to hear about.  You constantly hear about problems in the Quagga base code.  Even though they're fixing them, they're still out there.  So I wouldn't recommend it.  So OpenBGPD, instead, is native on OpenBSD.  And remember we said the shop was freeBSD based at the time.  And at the time the port was lagging behind inversions on freeBSD.  And the here present mic, he's denying it now, but I asked him about, he was using OpenBSD at the time.  OpenBGPD knows OpenBSD.  And he reported me that in his network he saw string things happening with OpenBSD.  So even though I was already using it in MENA, it's an internet exchange we have in Italy that I founded along with four other people,  I didn't feel comfortable using it because of this records, because of some tests that I made internally,  which didn't quite give me the expected results.  OpenBGPD is solid.  OpenBGPD is solid, yes.  OpenBSD is less solid and sometimes it will lose state of the interface and start,  you would unplug an interface and not see it going down in your internal IGP.  BERT was mature enough at the time, promising enough to be able to handle MPLS and multiple tables.  We were already using it at MENA, we were using both BERT and OpenBGPD.  It also supports OSPF and GRIP, so this is why we ended up using GRIP.  I don't know if one of my guys said this is okay, this can't put that good on the desktop.  Might be, yes.  But so far I haven't seen any problems.  You haven't laughed, so I mean...  No, well, even with...  Well, yesterday was two days ago, we had some major flapping on the Internet,  like six million route changes in a few hours.  Yeah, I didn't even notice.  So just for you to know, there were reports in the last few days of instability on the Internet,  so you would get lots and lots and lots and lots of updates.  So withdraw routes, add routes, withdraw routes, add routes, and this makes your CPU usage go weary  when this happens and six million changes in a few hours is a lot.  So we end up using BERT.  ExabGP instead is not really a BGPDmon, it's just a toolkit.  The idea was to use it for the CDN, so to do anycast and also to play with communities and backhauling, blackhauling.  So what ExabGP does is it runs as a very, very, very tiny demon where you can originate announcements  and you can accept announcements and do whatever you want with them, you can script it.  So we ended up writing a solution where we are doing blackhauling.  So this means that we inject a route into our IGP where ExabGP redistributes it into our border routers  and with a specific community telling our border routers to null route this specific IP  so that if we are getting, I don't know, scanned from one of those IPs, we just blackhaul that  and we don't receive any traffic anymore and after some time we just bring it back into, we cancel this route specifically.  So ExabGP works fairly well, you can also use it to test your BGP implementations or to test your routers  because you can inject even the full routing table and have it perpetrated into your system.  So let's go on with XDSL aggregation.  The goal was to do it in different countries, we're actually doing it in three.  One started it even across countries so every major incumbent you can find will do L2TP  so L2TP with PPPoE is a termination.  I guess they also do it in Canada, right?  So this was quite easy, there's MPD in FribuC, it's easy, simple and stable, never had any problem with it.  A configuration is very, very simple and if you need any help, Alexander is very, very, very, very, is available.  So you can just send, shoot him an email if you have any problem with the code or anything.  We actually had problems with some parts of the code, we emailed him, came back with an answer in less than 24 hours  and then we worked together to fix the problem and we came up with a good solution.  MPD can be interfaced easily with Radius so if you have users in Radius, you can use it.  You can have it look up in Radius, we also have a modification to MPD, we modified a little bit of the code to do some,  to tunnel again L2TP to continue, to carry on the tunnel towards some of our customers because we have,  at one point we started having resellers, so we, sorry?  No, it doesn't do tunnel switch.  So we got the call in from, open up the L2TP tunnel, we do, we look into the PPPOE session,  get the username and domain based on the domain, we forward it to another,  well, but we do it dynamically looking at announcements with CVBGP,  what we have in the PPPOE table from this customer before that to see a set of other service they wanted to do it.  They said we have it already working with other implementations in their French, so I won't get into any more details.  So other stuff you might need, these are tips from somebody who has run an ISP for some time.  So you want an IP address management or config manager, a looking glass, a NetFlow, anyone?  Do you do a NetFlow? No.  Sorry?  We didn't have an X-ray.  You want some recommend management software and diagnosis tools?  Sure.  Yeah, that's into, that's actually, that's what recommend management means.  I don't understand it, it's incredible to see how many times it has people with RACS servers and only a fellow server with login with the justice team.  Yeah, well the idea is to label everything.  People who have bigger operations might already know all of this, but recommend management is one side,  what I, how I called this.  There are different softwares, there is, well, IPAM and config manager, I ended up writing my own internal solution,  even for RAC management and cable management.  So, yeah, I think that's the next section, Simon put some slides of some disasters,  some interesting examples of how that works.  I actually had a ring label that I don't know where everything is, I don't have to get it, so we'll explore and find everything.  Yeah, well, the thing is when you're a bigger shop, you already have something in place for this, but since we were actually,  It's something that's found in the dump, it's far over, it's not your black community.  Yeah, exactly, exactly.  So what we, the problem here was that we already had some servers scattered all over and we started adding more, more equipment to do the BGP part and the DSL aggregation.  So we started needing something like this and when you, when you start adding transit, so two transits per, per pop in different countries,  then you want to start having something, some way to know where exactly where things are.  Yeah, of course it happens that you call the colo and have the wrong machine rebooted, but that still happens, but it should not.  Sorry?  It does, it will happen.  I recommend subnet's manager because it does both IPv4 and IPv6, it's very easy, it's written PHP, sorry, I didn't write it.  And if you live in the error in region, it can do, it can already send the, the swips directly to error in to register your assignments,  your assignments to your customers. I'm writing to integrate the ripe way of doing things in the, in the software, but not yet.  Do you guys know what the looking glass is?  The, looking glass is, the purpose of looking glass is to be, to look into what's happening in your routers or outside of your routers.  If you are, LG is the de facto standard but needs remote console access, so that means that it's based on Cisco, Cisco gear.  So it has to connect to the, to the console on the Cisco gear and run comments.  And if you're not running Cisco gear, you can't run it.  So GI XLG is a reasonable solution, it's based on XBGP, so that means that you might want to have XBGP running on the same machine as your, as your other,  it's based on GX, it's based on XBGP and uses MySQL as back end, so you can do, you can have XBGP put all the routes in, in MySQL and then look them up.  NetFlow, PF flow D, I would have liked to see Henning do it, because he committed it a few years ago and then never touched it again.  I don't know if it works, I never tried it, so actually for NetFlow there's not much going on, on, on the PSDs.  So if you have any hints or ideas, I would be glad to hear about them.  And, sorry?  Five years ago, I took the one in NetFlow.  No.  So, various others are, for example, rack management. My advice is to use RackMonkey if you don't want to rack your own, because it's great.  It lacks actually the ability to, to track cabling, but you can easily, you can easily manage it via some of the,  the notes, the notes system it has or you can manage, you can manage it. It's still in RackMonkey. It's not as, it's not as good as it would do if it was written for that purpose, but still you can do it.  Smoking, because when a customer calls in and says, oh, my connection is not working, you want to tell them I see that it is working or not.  MRTG cacti, depending on size or your choice, whatever system you want.  So now, I think it's Amadeo's time to go into details about the content delivery network. Let's see if that works.  Can you hear him?  Yes.  Can you guys see him or want me to put him on the, on the Vix screen?  Hello?  Can everybody, can you hear me?  Yeah, they can hear you.  Can you hear me?  Yes, we can hear you. Just go on.  Okay.  And I think we have a problem with the audio, I don't know, so if you need to tell me anything, use the keyboard.  Tell me here and what you think.  And I think we have a problem with the audio, I don't know, so if you need to tell me anything, please use the keyboard.  Okay.  Okay.  So, while we had this ISP in place with multiple locations, we also thought of using it to...  So, we reflected.  So we've done a VSD-based...  While we had this ISP in place with these multiple locations, it's right on the SCAOS screen too.  Yeah, we've done a VSD-based...  Hold on, hold on.  In this context, we know it can't.  So we do everything, we do all the information, we're just reading, and then the VSD-based...  Sorry? ...is made...  Yeah, we tried last night.  Now, just a quick overview of what a CDN is.  A CDN is made essentially...  Okay.  Okay, just a second.  Okay, now it should be...  Okay, with the audio.  Now, a CDN is made out of basically two pieces, a CDI, which is the infrastructure, what actually delivers the beat.  And then a Condom Management System that allows you to decide what to deliver and how to deliver,  and to provide value-added services to the end user.  So, what we have, the first thing in network design, if you want to do a CDI today, you can ask yourself,  shall I make it or buy it?  When we started all this back in 2006, to buy it was not really an option,  because there were very, very few players, and they were sort of extremely expensive,  and with little, if any, flexibility.  So, it was not really an option to buy the Condom Delivery infrastructure from someone else.  Today it is.  You might have a look at Cloudflare, for example.  Also, there is some other services from Amazon, at least for the basic delivery service.  I think if you want to do a CDN as a business, so if your core business is CDN, you need to run your own CDI.  Otherwise, you are just a sort of reseller for someone else's services.  So, you have different options if you want to make your own CDI.  Go for a single server, now this is of course an option, but it is not a real one, because the real reason to have CDIs  is to reduce the number of hopes and to deliver the bits as close as possible to the end users.  Also, you might consider a single delivery center, which is more or less the same thing.  So, don't go for it. What we do, we have multiple delivery centers, and that's what we will talk about.  So, okay.  The first thing you need to find a way, a technical way to make sure that the end users can reach the closest available version  of the content that they are looking for.  And so, you have different options.  You can use ITER and IP routing, so that using routing, BGP basically, you can direct the user to the closest available servers,  or you can use DNS.  We've been using both, and definitely the routing option is the best one.  Everything again, based on a BSD solution.  The second element that you need, you're doing really a network over the network,  because you need to be able to reach and to deliver similar content across your network, your servers.  So, you can ITER do it, roll to say so, so you go on the internet,  or maybe for some services, you might need an open VPN or an IPsec solution.  Beware of layer 2 or level 3 services that you might buy around, they call it VPN as well.  I do not recommend layer 3 services, because they don't add anything.  They might also be slower or less reliable than a standard internet connection.  We use open VPN to send, for example, accounting information or content localization.  You know the location.  You need these because you need to be able to have local information.  I mean that, and also local content.  Every distribution center needs to know everything about all the content that needs to be distributed,  where it is, how to locate it, and also need to have the information about accounting, for example.  If you want to disable a certain user or certain content,  the information from our point of view needs to be locally present and replicated and distributed in every distribution center.  You don't want to keep it centralized.  Because of, of course, if you have a failure somewhere, you need to be able to continue the service.  Now, for content delivery infrastructure, you need software.  The hardest part is to find a streaming solution, a video and audio streaming solution.  Today, you basically have three options. You can read them here.  Red 5, Wozze and Adobe Flash, Flash Media Server.  They're all Java based, so you can run them on any standard BSD platform with Java enabled.  Red 5 is open source and free.  On the other hand, it only provides delivery to RTMP, which means Flash devices.  The other two are more or less the same thing.  Adobe, possibly a little more expensive, and Wozze is the one we use because it's not that expensive,  and it offers a higher number of delivery options for recovery.  You can stream to mobile devices and like.  HTTP, if you need to deliver HTTP that's very basic, you use your preferred web server.  Maybe you need to write your own, I mean, both cases for streaming and HTTP,  you need to write your own accounting solution.  When you run a CDN, basically you sell bits, so you need to make sure you count how many bits go out to be your clients.  And also, you need to write plugins for the streaming services or application or whatever you call them,  to make sure you're able to locate the content, which server and which distribution center it is in.  And then to provide a number of value added servers, we'll come back later about the value added services that you can do.  Coming to the content management system, many people say, okay, that's the easy part.  If you've done the content-ravelling infrastructure, content management is the easy part.  I don't agree completely and it says that it is easy if you want to make just the basic things.  It can become a little bit challenging if you want to deliver especially for the value added services that we'll see in a moment.  Anyhow, you need to choose your web development platform because CMS is usually based on the web.  You might be willing to write an API via web services or whatever other solution you want to use  to integrate with your content-delivery infrastructure.  You need to keep an eye on accounting so that accounting information is available via the content management system.  And then you need to handle publishing.  When we talk about video streaming, usually people have the idea of a sort of YouTube  where you automatically, whenever you upload something, it's automatically you can view it.  Everybody can view it. You just copy and paste.  You are out to your friend and they can see the video.  Now, we decided to split a little bit the publishing from the uploading.  So you're responsible to publish the content on your own.  This is a sort of big difference because we focus mainly on B2C market.  We don't go directly to the end user.  So you want, for example, to be able to have a wide label services that you want to rebrand.  Or maybe you want to show that your company is now a relationship with our streaming platform.  So the publishing is left aside.  The value-added services, here I mentioned a few, you can do multi-conference with streaming.  Streaming can be only directional as it is now, or it can be also bidirectional.  Now, if you multiply with many users, you can do multi-conference.  It's a little bit tricky. It's not that easy, but still it's possible to do it.  For example, also you can do content tagging.  We've been using content tagging with a client or farce, and they have a lot of video,  so they need to be able to recognize them somehow.  As a scontrol, for example, let's say you want to allow only invited people to see your video.  And maybe content encryption, because maybe you want not your delivery sensible videos,  for example, meetings or things like that, and you want to use encryption.  Now, one of the toughest part, and with this I'm about to complete this short speech about streaming,  how to do content preparation.  Now, content preparation is when you have videos, or I mean generally speaking when you have content,  but the hard part is to deal with video and audio.  So, the minimum toolset that you need to use is GIPPAC, FFMPEG, MPLayer and Gstreamer, which is not mentioned here.  You want to be an expert with these tools, because you always need to convert, transform.  There is no real defined standards in the video streaming industry, and many new standards pop up continuously.  So, you need to sort of convert and make sure you can transcode or adapt the video for video streaming.  Now, if you use FreeBSD, you might be willing to have a look at user ports multimedia and user ports audio.  There is a lot of stuff that is very useful.  And then, if you want to do cool stuff with videos, you should really be using user ports multimedia MLT.  This allows you to write scripts that can add effects to a video, merge videos together,  show things, popping up and down in video, change the audio, make effects and the like.  So, it's really powerful tool.  And the last slide is about the script that is running this conference.  It's running on BSD, uses standard stuff.  We've been using a camera connected to a laptop with a firework cable.  I'm making a copy of it, so you can see the T-Common.  I'm making a copy of the video, so if I want to work and use it afterward.  And then, I'm encoding live with FFMPEG directly into our ingest point.  And that's it. Now, should you have any questions, please ask them to Massimiliano and Massimiliano.  Any questions?  How do you guys handle the broadcasting? I'm hearing that you use this same method.  Wait, somebody else.  How?  You need to define the edge, because the edge of the internet is, as far as I understand, not a unique place.  It depends on how you define the edge.  The edge is where we have a content delivery, content distribution center.  That's what we call the edge.  Now, we don't have a centralized structure.  So, for example, when you upload a video on the...  Yes, okay. When you upload a video via the content management system, it is automatically distributed to all the content distribution centers.  Now, what happens if the content is not requested for a period of time, it might get deleted on some content distribution centers.  And then, we know how to locate it.  And if it is requested again, we copy it again, we send it again to the content distribution system.  And how do we handle this? This is something we had to write and we had to do on our own.  It's based on FuseFS.  We're going with some final considerations which are mostly related to the networking part, a little less about the streaming part.  So, beware.  While it might be easy to set up, you have to maintain this kind of environment, which means you need to have knowledgeable people.  One of the challenges that I found at my previous job was that I had some employees who were networking guys.  And I told them, okay, let's implement this software routing system.  And they were scared about it because they didn't know where to start from.  Because they had never seen a Unix console or whatever.  And so, that was a major challenge in trying to implement this.  And this is also one of the reasons why I had to...  Well, this is one of the reasons why at the end I decided just to leave.  Yeah, I'll leave and sell my shares of the company.  So, many networkers know Cisco and Juniper and maybe a little bit of MicroTik, but they don't know Unix.  They don't have the knowledge or the need to get to know that.  Because if you tell them to go learn this, they will just switch to another job.  And this is something you want to keep in mind.  And VGP is a threat.  Well, you could be a threat to the internet.  So, you have to know what you're doing when dealing with VGP.  So, who designed it?  Whoever designed it thought that you have to be knowledgeable to run it.  You can do a great deal of damage to the whole internet when you're running VGP.  So, take for example the YouTube against Pakistan Telecom incident.  Somebody in Pakistan started to announce the YouTube prefixes.  And so, all the traffic directed to YouTube went to Pakistan instead of going to YouTube directly.  And there are newbies playing around with MicroTik.  Instead of doing, they created an ISPath of 256 entries.  And they tickled the bug in all the other Cisco routers all over the world.  And sessions all over the world were flopping around.  And it took a while to figure that out for the whole internet.  So, you know, you can't just email going back and forth to between people.  Sorry?  They were newbies in MicroTik.  And didn't want to run, didn't want to read the manual.  So, and so on.  There are incidents like these pretty much every day.  Some have a smaller influence on the internet, some have a bigger one.  And yes, we showed it as possible.  Now it's up to you.  And one of the things I'd like to highlight is if you run, if you have a network,  if you have an ISP, you want one of these probes to be in your network.  Because there, this is a project from the RIPE NCC to monitor the internet.  So, if you want one, just let me know.  And I can ship you one from my office.  So, you can install one of these probes at home in your office network.  And you can monitor the, you can use them to monitor the whole internet from there.  Or you can, you can use, you can use a, you can, you can play with one of these probes  and get points using them to run your own test measurements against your own network.  So, if you want to say, I have 500 probes all over the world,  ping one of your IP addresses to see the reachability.  You can do that from the, from the Atlas website.  Or you can, you can run any, any other kind of measurement.  So, it's, for now you can only do ping trace route and DNS lookup.  But it's going to become a, it's a toolbox to run your own measurements on.  World effort worldwide.  Do you have any other questions?  No questions?  We're either.  I just, looking at some of the technologies that you're using.  And just from what I've read.  Yeah.  And it seems like a lot of the technologies you're using are some, not most,  are the same that are integrated into ISP today, and like, correct,  with the exception of somebody obviously, the content of the pre-infrastructure component.  You mean for the, for the streaming part?  Well, ISP config is more for, it's more suitable for hosting, hosting sites.  So that, I didn't get into that detail because I was mostly focusing on the access part.  But yes, some of, most of that is, well, I didn't get into the web server side of the,  of the operations, but web server or email or whatever else side of the operations.  But yes.  What do you do with the accounting, the DNS?  Yes, well, well, DNS, when you do access you want to just generally manage only reverse  DNS for your customers.  And again, I ended up writing my, my own software.  It was actually integrated into the IP address management software where customers could  just log in and see their IP addresses and set up the reverse DNS.  It's trivial.  Very, very, very trivial.  So, yes, I didn't get into that detail because I was focusing more on the routing part.  But yes, it's, it's mostly standard software.  Yeah.  So, yes, I'm, well, BGP is, why do they have that?  So, yes.  So, for the rest is pretty much common.  Any other question?  Do you build a time on the transit?  Yeah.  No, not the transit.  A transit we do on antico-person-tile to, well, we have some co-location customers and  we do on antico-person-tile on them.  How do you build a time on that?  MRTG.  Sorry?  Yeah.  We're not very strict because actually, as, again, we go back to the content versus eyeballs,  content and eyeballs.  If you, if you keep a good balance between outgoing and incoming bandwidth, you don't want to care  about what your customers are really doing.  I mean, well, you still have to keep an eye on that, but you're not so strict.  Unless you really want to charge them based on each and every megabit that they're doing,  there's no reason to do that.  So far, we do it just look at the graphs at the end of the month and that's it.  Yeah.  Yeah.  Yeah.  Yeah.  Same thing.  Unless, unless they're, they, they bought like 10 megabits of transit and then they do  20 or 25, well, then yes.  They generally send them an email telling them, hey, this month you did this person pays  more than you should be allowed.  Are you guys comfortable with the 95th person tell how it works?  No.  Okay.  So basically you run MRTG and you, and you, you get a sample of the traffic every five  minutes.  So you, at that point, you know how much traffic has been passing by on that, on that interface.  And at the end of the month, you discard the highest 5% of those, those readings and then  you get the average out of that.  So you see that on, on average, your customer has done 20 megabits and you discard the 5%  higher readings.  So that means that he can do for 36 hours a month, he can do say if he bought 10 megabits  of transit over 100 megabits port, up in port, they can do 100 megabits for 36 hours a month.  Which means that if they have spikes or if they, if they want to do, I don't know, advertisement  campaigns, they can still do them and not be built for them.  And while you, while you might have big, big spikes on your network, you can still be sure  that if they do more, if they use more traffic, then they will get built, or you can build  it.  Sorry?  Exactly.  Easy to put into this, you get the old, the old, the old 30 megabits.  So.  Quite a long, this may already be 30 hours.  But what, what generally happens is that, I know some, some of the customers will pay  20 megabits and then the 95th person to be able to do 21 on average.  You don't want to start billing them for 21.  You just play the good, the good part and send them an email and say, hey, you know, you've  been doing 21 megabits and then you might think about getting 25 of transit instead of just  20.  And by the happens rarely because, well, either you are proactive or when you have both  content and, and access on your, on the same network, you can, if you start having a lot  of access, you can start actually giving content bandwidth away for free.  You can start playing with it like using it as a commodity to encourage customers to stay  with you.  Sorry?  Yeah, some costs because you're, you're always paid for, for bandwidth in both directions.  So you will, if you're, if you're having access and you have 30 megabits of traffic for your  access network, you will have 30 megabits getting to your network.  And then you will still not, you might not still be sending too many bits out of your  network.  So if you, if you get co-location customers and you, and they will start sending bits out  of your network, that's bandwidth you're already paying for.  And then you can start playing with it from a sales perspective, sounding it for very  cheap.  Well, there's a case here in Ontario, Canada, there's a very large business, a very large  business model,  I mean, if anyone supported around code, many did second-hand code pump from 2007,  Thank you.
The_BSD_ISP_0_767.png The_BSD_ISP_768_1187.png The_BSD_ISP_1188_1955.png The_BSD_ISP_1956_2267.png The_BSD_ISP_2268_3719.png The_BSD_ISP_3720_3815.png The_BSD_ISP_3816_4403.png The_BSD_ISP_4404_4475.png The_BSD_ISP_4476_5555.png The_BSD_ISP_5556_5855.png The_BSD_ISP_5856_6035.png The_BSD_ISP_6036_6107.png The_BSD_ISP_6108_6803.png The_BSD_ISP_6804_7499.png The_BSD_ISP_7500_7583.png The_BSD_ISP_7584_9707.png The_BSD_ISP_9708_10523.png The_BSD_ISP_10524_12131.png The_BSD_ISP_12132_18083.png The_BSD_ISP_18084_18611.png The_BSD_ISP_18612_19295.png The_BSD_ISP_19296_20711.png The_BSD_ISP_20712_22163.png The_BSD_ISP_22164_22343.png The_BSD_ISP_22344_23339.png The_BSD_ISP_23340_24779.png The_BSD_ISP_24780_27371.png The_BSD_ISP_27372_29711.png The_BSD_ISP_29712_29819.png The_BSD_ISP_29820_29915.png The_BSD_ISP_29916_30011.png The_BSD_ISP_30012_30131.png The_BSD_ISP_30132_30383.png The_BSD_ISP_30384_33299.png The_BSD_ISP_33300_34115.png The_BSD_ISP_34116_34259.png The_BSD_ISP_34260_35135.png The_BSD_ISP_35136_38579.png The_BSD_ISP_38580_40115.png The_BSD_ISP_40116_40703.png The_BSD_ISP_40704_42251.png The_BSD_ISP_42252_42935.png The_BSD_ISP_42936_44087.png The_BSD_ISP_44088_44171.png The_BSD_ISP_44172_44219.png The_BSD_ISP_44220_44267.png The_BSD_ISP_44268_44315.png The_BSD_ISP_44316_44375.png The_BSD_ISP_44376_44543.png The_BSD_ISP_44544_44639.png The_BSD_ISP_44640_44759.png The_BSD_ISP_44760_44903.png The_BSD_ISP_44904_44975.png The_BSD_ISP_44976_45455.png The_BSD_ISP_45456_45635.png The_BSD_ISP_45636_45911.png The_BSD_ISP_45912_47027.png The_BSD_ISP_47028_49319.png The_BSD_ISP_49320_49403.png The_BSD_ISP_49404_53687.png The_BSD_ISP_53688_56843.png The_BSD_ISP_56844_61871.png The_BSD_ISP_61872_64559.png The_BSD_ISP_64560_65987.png The_BSD_ISP_65988_70295.png The_BSD_ISP_70296_72143.png The_BSD_ISP_72144_74363.png The_BSD_ISP_74364_76163.png The_BSD_ISP_76164_76811.png The_BSD_ISP_76812_78167.png The_BSD_ISP_78168_78635.png The_BSD_ISP_78636_79751.png The_BSD_ISP_79752_80507.png The_BSD_ISP_80508_81131.png The_BSD_ISP_81132_81251.png The_BSD_ISP_81252_81647.png The_BSD_ISP_81648_82379.png The_BSD_ISP_82380_83423.png The_BSD_ISP_83424_84887.png The_BSD_ISP_84888_85007.png The_BSD_ISP_85008_85319.png The_BSD_ISP_85320_85475.png The_BSD_ISP_85476_85751.png The_BSD_ISP_85752_86435.png The_BSD_ISP_86436_86483.png The_BSD_ISP_86484_86709.png
Just one person. How many people? Two people. Three people. Y'all, you are sort of an ISP. Do you still work at ISC? Okay. How many of you do have experience with BGP? Okay. What kind of experience? Like running, configuring, public, no... Man, school. Sorry? School. Okay. Well, that is different from the real world. Well, agenda. So we have an intro background. We'll see how the background of what we've been doing over the last few years will go into details about some tools that we're using. And then we'll talk about the streaming services. So who are we? I am a RIPAA NCC trainer. I teach people about how to use IP addresses in the European region. I am a past founder of an ISP. So I created it from scratch. Everything configured from the first line of code from the first configuration line in our outer to... We were configuring bank networks. And I'm involved in networking some other projects. And Amadeo is the owner and operator right now of Bstream. He also founded the ISP 1997, the first fully automated ISP in Italy running on VSD. He's a coder. You can see him. If you go to this website, there's a live stream directly from Amadeo because he couldn't come over. So if you type it in your web browser, you should see a live stream from him. Wait a few seconds to type it. Sorry? Are you using Capitals? No. Well, he should be on one. I can see him. Can you see other folks in the wall? Yeah, yeah, it's coming. There he is. It's coming. There he is. Okay. I got that domain market.com, that stream.com. That stream. That's the original. I guess that's a totally different kind of thing. I don't know. You can do that. How do you watch the live stream? Well, later. So what is this about? The idea is to open up discussion for people to use open source software to drive their business. In the networking world, when you start mentioning, oh, I run a software router and Mike knows about that. People look at you and they say, ah, sometimes you have even problems bringing your own hardware into IXPs because they don't want that. They don't know what it is and it's a problem. I found this to be a problem in my previous job. So what I'm trying to do, what I generally try to do is to get people to start using software solutions to overcome problems. And we'll see why then. But so I'd like you to participate. So if you have questions, if you want to give me insights from your, or if you want to just raise your hand, talk, participate, and you can also set up an hashtag on Twitter. And there you go. So why are we doing this? Because I've seen it works. And because we're going to show our experience with it, well, some of it. And ultimately, because it's definitely cheaper than other solutions. Definitely cheaper, depending on how you look at it. If you look at it from the perspective, from the real money, on the money side, yes, if you look on the operational side, you have to train people and we'll see that later. What is the background? Streaming is a streaming company. It is based on FreeBSD. And for streaming, especially live, you need connectivity. So we were, me and Amadeo have been friends for many years. We were discussing about this after I left my previous company. And he was telling me that he was having problems finding good connectivity in Europe for some of the streams he was supposed to do. And then I came up with an idea. Let's try to become an exercise speed. We won't be discussing about the paperwork here. There's a lot of paperwork involved in Europe to become an operator, to get licenses. It's different from getting a license from the FCC in the US, but it's still hard. And it takes a lot of effort. But it can become a major selling point when you're trying to go to one of your customers, selling them a streaming service, and you come up with a complete solution that will bring you the connection service, will bring you the access, and will also give you the stream. I don't know how you use it, but I'm doing what I do with this. You don't deal with... Okay, but how do you deal with this when you go to your customers, when you're asked to run a live streaming or... Sorry? So you don't do that? Okay, why not? It's not in your business plans? I'm not in the business plans. Okay, so you don't deal with it? Okay, perfect. Sorry for the question. What were the goals? Provide access along content. Keep the costs low. So we said, okay, let's try it. Let's try it if it works, and then if it works, we'll get bigger hardware. We'll get more, and we wanted to use technology which could grow with, because the problem, and we will see later, is that some of the technology you can buy nowadays will be obsolete in a very short period of time. We wanted to mix up content and eyeballs. When you do streaming and you're sending lots of bits across the networks, you're sending them out of your network. But when you start doing access, you will begin to have traffic coming into your network, and so you will have a balanced... The goal was to get a balanced ratio between outgoing and incoming traffic, so that when you go to PIRIT on IXP, you will get more attention from the other networks, because they won't have to deal with... Well, how can I put this? They will more likely PIR with you than when you're just a content network, and you will have also eyeballs for their contents, so if you have a balanced ratio of traffic, it will be easier for them to deal with you. How would you achieve that balanced ratio of traffic if you have to start small and you have to start somewhere? So really you're starting with content delivery now, right? And then as your clients grow, you're saying then, is it your staff to plan PIRIT arrangements with other upstream... Okay, no, the problem is when you do this, you act with hot potato routing, so what you will do is you will hand out your bits, your content bits to the closest point of PIR with any other network, and then they'll have to deal with bringing your bits into their network and bringing them to their users, and that's a major deal. So you're sending more than receiving, and this is a cost for them, but when you have a balanced ratio, you will see you as a point where they can send you bits as well. And so generally, let's say you get to a new internet exchange and you send out an email, hey, I want to PIR with you guys. What generally the answers you will get are, smaller networks will say, yes, okay, we'll PIR with you. No problem, we'll set it up, but the amount of traffic will be very low. Bigger networks tend to tell you, hey, yes, we could PIR with you if you meet these requirements. And the requirements generally are, I don't know, one gigabit of traffic, symmetrical, so not just in one direction, PIR over multiple internet exchanges in different regions of the world. So sometimes you have to be big to PIR with them. Sometimes you can just try and tell them, hey, I have, I don't know, three gigabits of traffic, but in just one region, would you PIR with me? Because I see you have eyeballs or you have content. So generally it helps if you, well, we'll see this later, if you generate graphs based on net flow output and look at the destination and origination ASNs for the traffic you generate or receive, and you show them, hey, I have two gigabits of traffic coming from your network into mine, but I don't have any traffic coming out of my network into yours, would you like to PIR with me at this internet exchange? So other than passing through forensics, we will just exchange traffic this way. So it's a matter of giving them a chance to see the value in managing a PIR session with you. Because remember, if you're a big network, you will have thousands of sessions to keep up unless you find internet exchanges with good and valuable route servers, which is a different kind of story. Do you guys know what a route server is at an internet exchange? I don't see many ads shaking. Okay, so, well, Mike knows. If you go to an internet exchange instead of PIRing with all the different entities, you will PIR with just one machine, one session, and you'll get all the routes from all the other participants, and then you will send traffic directly to their interfaces. So instead of having, say, 200 sessions, you will just have one, but you will receive routes for all the other participants, which makes life easier. For example, Google only PIRs at internet exchanges with a route server, and otherwise they would have to manage thousands and thousands of sessions all the time. And you turn the drive to the internet. I don't know about that. I don't know the internals, but we're not here to discuss the internals. That was just an example. So, does this answer your question? Is there any other questions so far? No. So the first steps are define a routing policy, define Xcon, and cross connects for XDSL services and related protocols and standards, because there are a bunch of them, define new goals for the CDI content delivery infrastructure, and shop for vendors and or software. So, routing, how do you think about doing it? BGP, of course. Maybe OSPF as IGP, although we ended up with a different solution with RIP for the moment, and as you can see from my t-shirt, I wanted to do IPv6 from the beginning. So the idea was to begin with IPv6 and then do IPv4 on top of that. What are the routing challenges nowadays? The routing table is continuously growing. It's almost touching 500,000 routes. We are 556,000 at the end of this week. We at the beginning was 430,000. You have to be IPv6 enabled if you want to be future proof. And if you want to be IPv6 enabled, well, there will be more routes to come. And it will be fun. I mean, I'm really wondering what will happen when people will start announcing Slash48 separately, because this will mean that any provider on the world might be able to announce up to 65,000 different routes. So I imagine you multiply it by the number of ASN numbers and the number of networks that are out there. So the problem is with vendors, many of them, they have T-cams, so the memory on the routers, is so far limited on the low-end models to 1 million IPv4 routes. So if you have more than... If the routing table goes to more than 500,000 entries and you have more than one transit, you won't be able to store the whole routing table on your router. So the problem is it will start to save them memory and it will be slower and slower and slower in loop backups. And add to this the fact that you also want to have IPv6 routes. So you will also have even more and then your machine will just slow down. Unless you want to cash out money for higher-end equipment, which means, of course, getting more routes and getting more money out of your pockets. So how do you solve this problem? You can use Quagga, OpenBGPD, BERT, and ExaBGP. Quagga is the closer to the Cisco implementation. Quagga has a DemonRunning and then you have an interface which is very similar to the Cisco interface. So if you have somebody who is familiar with that, it's very easy to move them from a pure Cisco router to a Quagga-based router. But it has way many more bugs than I would have liked to know and see or to hear about. You constantly hear about problems in the Quagga base code. Even though they're fixing them, they're still out there. So I wouldn't recommend it. So OpenBGPD, instead, is native on OpenBSD. And remember we said the shop was freeBSD based at the time. And at the time the port was lagging behind inversions on freeBSD. And the here present mic, he's denying it now, but I asked him about, he was using OpenBSD at the time. OpenBGPD knows OpenBSD. And he reported me that in his network he saw string things happening with OpenBSD. So even though I was already using it in MENA, it's an internet exchange we have in Italy that I founded along with four other people, I didn't feel comfortable using it because of this records, because of some tests that I made internally, which didn't quite give me the expected results. OpenBGPD is solid. OpenBGPD is solid, yes. OpenBSD is less solid and sometimes it will lose state of the interface and start, you would unplug an interface and not see it going down in your internal IGP. BERT was mature enough at the time, promising enough to be able to handle MPLS and multiple tables. We were already using it at MENA, we were using both BERT and OpenBGPD. It also supports OSPF and GRIP, so this is why we ended up using GRIP. I don't know if one of my guys said this is okay, this can't put that good on the desktop. Might be, yes. But so far I haven't seen any problems. You haven't laughed, so I mean... No, well, even with... Well, yesterday was two days ago, we had some major flapping on the Internet, like six million route changes in a few hours. Yeah, I didn't even notice. So just for you to know, there were reports in the last few days of instability on the Internet, so you would get lots and lots and lots and lots of updates. So withdraw routes, add routes, withdraw routes, add routes, and this makes your CPU usage go weary when this happens and six million changes in a few hours is a lot. So we end up using BERT. ExabGP instead is not really a BGPDmon, it's just a toolkit. The idea was to use it for the CDN, so to do anycast and also to play with communities and backhauling, blackhauling. So what ExabGP does is it runs as a very, very, very tiny demon where you can originate announcements and you can accept announcements and do whatever you want with them, you can script it. So we ended up writing a solution where we are doing blackhauling. So this means that we inject a route into our IGP where ExabGP redistributes it into our border routers and with a specific community telling our border routers to null route this specific IP so that if we are getting, I don't know, scanned from one of those IPs, we just blackhaul that and we don't receive any traffic anymore and after some time we just bring it back into, we cancel this route specifically. So ExabGP works fairly well, you can also use it to test your BGP implementations or to test your routers because you can inject even the full routing table and have it perpetrated into your system. So let's go on with XDSL aggregation. The goal was to do it in different countries, we're actually doing it in three. One started it even across countries so every major incumbent you can find will do L2TP so L2TP with PPPoE is a termination. I guess they also do it in Canada, right? So this was quite easy, there's MPD in FribuC, it's easy, simple and stable, never had any problem with it. A configuration is very, very simple and if you need any help, Alexander is very, very, very, very, is available. So you can just send, shoot him an email if you have any problem with the code or anything. We actually had problems with some parts of the code, we emailed him, came back with an answer in less than 24 hours and then we worked together to fix the problem and we came up with a good solution. MPD can be interfaced easily with Radius so if you have users in Radius, you can use it. You can have it look up in Radius, we also have a modification to MPD, we modified a little bit of the code to do some, to tunnel again L2TP to continue, to carry on the tunnel towards some of our customers because we have, at one point we started having resellers, so we, sorry? No, it doesn't do tunnel switch. So we got the call in from, open up the L2TP tunnel, we do, we look into the PPPOE session, get the username and domain based on the domain, we forward it to another, well, but we do it dynamically looking at announcements with CVBGP, what we have in the PPPOE table from this customer before that to see a set of other service they wanted to do it. They said we have it already working with other implementations in their French, so I won't get into any more details. So other stuff you might need, these are tips from somebody who has run an ISP for some time. So you want an IP address management or config manager, a looking glass, a NetFlow, anyone? Do you do a NetFlow? No. Sorry? We didn't have an X-ray. You want some recommend management software and diagnosis tools? Sure. Yeah, that's into, that's actually, that's what recommend management means. I don't understand it, it's incredible to see how many times it has people with RACS servers and only a fellow server with login with the justice team. Yeah, well the idea is to label everything. People who have bigger operations might already know all of this, but recommend management is one side, what I, how I called this. There are different softwares, there is, well, IPAM and config manager, I ended up writing my own internal solution, even for RAC management and cable management. So, yeah, I think that's the next section, Simon put some slides of some disasters, some interesting examples of how that works. I actually had a ring label that I don't know where everything is, I don't have to get it, so we'll explore and find everything. Yeah, well, the thing is when you're a bigger shop, you already have something in place for this, but since we were actually, It's something that's found in the dump, it's far over, it's not your black community. Yeah, exactly, exactly. So what we, the problem here was that we already had some servers scattered all over and we started adding more, more equipment to do the BGP part and the DSL aggregation. So we started needing something like this and when you, when you start adding transit, so two transits per, per pop in different countries, then you want to start having something, some way to know where exactly where things are. Yeah, of course it happens that you call the colo and have the wrong machine rebooted, but that still happens, but it should not. Sorry? It does, it will happen. I recommend subnet's manager because it does both IPv4 and IPv6, it's very easy, it's written PHP, sorry, I didn't write it. And if you live in the error in region, it can do, it can already send the, the swips directly to error in to register your assignments, your assignments to your customers. I'm writing to integrate the ripe way of doing things in the, in the software, but not yet. Do you guys know what the looking glass is? The, looking glass is, the purpose of looking glass is to be, to look into what's happening in your routers or outside of your routers. If you are, LG is the de facto standard but needs remote console access, so that means that it's based on Cisco, Cisco gear. So it has to connect to the, to the console on the Cisco gear and run comments. And if you're not running Cisco gear, you can't run it. So GI XLG is a reasonable solution, it's based on XBGP, so that means that you might want to have XBGP running on the same machine as your, as your other, it's based on GX, it's based on XBGP and uses MySQL as back end, so you can do, you can have XBGP put all the routes in, in MySQL and then look them up. NetFlow, PF flow D, I would have liked to see Henning do it, because he committed it a few years ago and then never touched it again. I don't know if it works, I never tried it, so actually for NetFlow there's not much going on, on, on the PSDs. So if you have any hints or ideas, I would be glad to hear about them. And, sorry? Five years ago, I took the one in NetFlow. No. So, various others are, for example, rack management. My advice is to use RackMonkey if you don't want to rack your own, because it's great. It lacks actually the ability to, to track cabling, but you can easily, you can easily manage it via some of the, the notes, the notes system it has or you can manage, you can manage it. It's still in RackMonkey. It's not as, it's not as good as it would do if it was written for that purpose, but still you can do it. Smoking, because when a customer calls in and says, oh, my connection is not working, you want to tell them I see that it is working or not. MRTG cacti, depending on size or your choice, whatever system you want. So now, I think it's Amadeo's time to go into details about the content delivery network. Let's see if that works. Can you hear him? Yes. Can you guys see him or want me to put him on the, on the Vix screen? Hello? Can everybody, can you hear me? Yeah, they can hear you. Can you hear me? Yes, we can hear you. Just go on. Okay. And I think we have a problem with the audio, I don't know, so if you need to tell me anything, use the keyboard. Tell me here and what you think. And I think we have a problem with the audio, I don't know, so if you need to tell me anything, please use the keyboard. Okay. Okay. So, while we had this ISP in place with multiple locations, we also thought of using it to... So, we reflected. So we've done a VSD-based... While we had this ISP in place with these multiple locations, it's right on the SCAOS screen too. Yeah, we've done a VSD-based... Hold on, hold on. In this context, we know it can't. So we do everything, we do all the information, we're just reading, and then the VSD-based... Sorry?...is made... Yeah, we tried last night. Now, just a quick overview of what a CDN is. A CDN is made essentially... Okay. Okay, just a second. Okay, now it should be... Okay, with the audio. Now, a CDN is made out of basically two pieces, a CDI, which is the infrastructure, what actually delivers the beat. And then a Condom Management System that allows you to decide what to deliver and how to deliver, and to provide value-added services to the end user. So, what we have, the first thing in network design, if you want to do a CDI today, you can ask yourself, shall I make it or buy it? When we started all this back in 2006, to buy it was not really an option, because there were very, very few players, and they were sort of extremely expensive, and with little, if any, flexibility. So, it was not really an option to buy the Condom Delivery infrastructure from someone else. Today it is. You might have a look at Cloudflare, for example. Also, there is some other services from Amazon, at least for the basic delivery service. I think if you want to do a CDN as a business, so if your core business is CDN, you need to run your own CDI. Otherwise, you are just a sort of reseller for someone else's services. So, you have different options if you want to make your own CDI. Go for a single server, now this is of course an option, but it is not a real one, because the real reason to have CDIs is to reduce the number of hopes and to deliver the bits as close as possible to the end users. Also, you might consider a single delivery center, which is more or less the same thing. So, don't go for it. What we do, we have multiple delivery centers, and that's what we will talk about. So, okay. The first thing you need to find a way, a technical way to make sure that the end users can reach the closest available version of the content that they are looking for. And so, you have different options. You can use ITER and IP routing, so that using routing, BGP basically, you can direct the user to the closest available servers, or you can use DNS. We've been using both, and definitely the routing option is the best one. Everything again, based on a BSD solution. The second element that you need, you're doing really a network over the network, because you need to be able to reach and to deliver similar content across your network, your servers. So, you can ITER do it, roll to say so, so you go on the internet, or maybe for some services, you might need an open VPN or an IPsec solution. Beware of layer 2 or level 3 services that you might buy around, they call it VPN as well. I do not recommend layer 3 services, because they don't add anything. They might also be slower or less reliable than a standard internet connection. We use open VPN to send, for example, accounting information or content localization. You know the location. You need these because you need to be able to have local information. I mean that, and also local content. Every distribution center needs to know everything about all the content that needs to be distributed, where it is, how to locate it, and also need to have the information about accounting, for example. If you want to disable a certain user or certain content, the information from our point of view needs to be locally present and replicated and distributed in every distribution center. You don't want to keep it centralized. Because of, of course, if you have a failure somewhere, you need to be able to continue the service. Now, for content delivery infrastructure, you need software. The hardest part is to find a streaming solution, a video and audio streaming solution. Today, you basically have three options. You can read them here. Red 5, Wozze and Adobe Flash, Flash Media Server. They're all Java based, so you can run them on any standard BSD platform with Java enabled. Red 5 is open source and free. On the other hand, it only provides delivery to RTMP, which means Flash devices. The other two are more or less the same thing. Adobe, possibly a little more expensive, and Wozze is the one we use because it's not that expensive, and it offers a higher number of delivery options for recovery. You can stream to mobile devices and like. HTTP, if you need to deliver HTTP that's very basic, you use your preferred web server. Maybe you need to write your own, I mean, both cases for streaming and HTTP, you need to write your own accounting solution. When you run a CDN, basically you sell bits, so you need to make sure you count how many bits go out to be your clients. And also, you need to write plugins for the streaming services or application or whatever you call them, to make sure you're able to locate the content, which server and which distribution center it is in. And then to provide a number of value added servers, we'll come back later about the value added services that you can do. Coming to the content management system, many people say, okay, that's the easy part. If you've done the content-ravelling infrastructure, content management is the easy part. I don't agree completely and it says that it is easy if you want to make just the basic things. It can become a little bit challenging if you want to deliver especially for the value added services that we'll see in a moment. Anyhow, you need to choose your web development platform because CMS is usually based on the web. You might be willing to write an API via web services or whatever other solution you want to use to integrate with your content-delivery infrastructure. You need to keep an eye on accounting so that accounting information is available via the content management system. And then you need to handle publishing. When we talk about video streaming, usually people have the idea of a sort of YouTube where you automatically, whenever you upload something, it's automatically you can view it. Everybody can view it. You just copy and paste. You are out to your friend and they can see the video. Now, we decided to split a little bit the publishing from the uploading. So you're responsible to publish the content on your own. This is a sort of big difference because we focus mainly on B2C market. We don't go directly to the end user. So you want, for example, to be able to have a wide label services that you want to rebrand. Or maybe you want to show that your company is now a relationship with our streaming platform. So the publishing is left aside. The value-added services, here I mentioned a few, you can do multi-conference with streaming. Streaming can be only directional as it is now, or it can be also bidirectional. Now, if you multiply with many users, you can do multi-conference. It's a little bit tricky. It's not that easy, but still it's possible to do it. For example, also you can do content tagging. We've been using content tagging with a client or farce, and they have a lot of video, so they need to be able to recognize them somehow. As a scontrol, for example, let's say you want to allow only invited people to see your video. And maybe content encryption, because maybe you want not your delivery sensible videos, for example, meetings or things like that, and you want to use encryption. Now, one of the toughest part, and with this I'm about to complete this short speech about streaming, how to do content preparation. Now, content preparation is when you have videos, or I mean generally speaking when you have content, but the hard part is to deal with video and audio. So, the minimum toolset that you need to use is GIPPAC, FFMPEG, MPLayer and Gstreamer, which is not mentioned here. You want to be an expert with these tools, because you always need to convert, transform. There is no real defined standards in the video streaming industry, and many new standards pop up continuously. So, you need to sort of convert and make sure you can transcode or adapt the video for video streaming. Now, if you use FreeBSD, you might be willing to have a look at user ports multimedia and user ports audio. There is a lot of stuff that is very useful. And then, if you want to do cool stuff with videos, you should really be using user ports multimedia MLT. This allows you to write scripts that can add effects to a video, merge videos together, show things, popping up and down in video, change the audio, make effects and the like. So, it's really powerful tool. And the last slide is about the script that is running this conference. It's running on BSD, uses standard stuff. We've been using a camera connected to a laptop with a firework cable. I'm making a copy of it, so you can see the T-Common. I'm making a copy of the video, so if I want to work and use it afterward. And then, I'm encoding live with FFMPEG directly into our ingest point. And that's it. Now, should you have any questions, please ask them to Massimiliano and Massimiliano. Any questions? How do you guys handle the broadcasting? I'm hearing that you use this same method. Wait, somebody else. How? You need to define the edge, because the edge of the internet is, as far as I understand, not a unique place. It depends on how you define the edge. The edge is where we have a content delivery, content distribution center. That's what we call the edge. Now, we don't have a centralized structure. So, for example, when you upload a video on the... Yes, okay. When you upload a video via the content management system, it is automatically distributed to all the content distribution centers. Now, what happens if the content is not requested for a period of time, it might get deleted on some content distribution centers. And then, we know how to locate it. And if it is requested again, we copy it again, we send it again to the content distribution system. And how do we handle this? This is something we had to write and we had to do on our own. It's based on FuseFS. We're going with some final considerations which are mostly related to the networking part, a little less about the streaming part. So, beware. While it might be easy to set up, you have to maintain this kind of environment, which means you need to have knowledgeable people. One of the challenges that I found at my previous job was that I had some employees who were networking guys. And I told them, okay, let's implement this software routing system. And they were scared about it because they didn't know where to start from. Because they had never seen a Unix console or whatever. And so, that was a major challenge in trying to implement this. And this is also one of the reasons why I had to... Well, this is one of the reasons why at the end I decided just to leave. Yeah, I'll leave and sell my shares of the company. So, many networkers know Cisco and Juniper and maybe a little bit of MicroTik, but they don't know Unix. They don't have the knowledge or the need to get to know that. Because if you tell them to go learn this, they will just switch to another job. And this is something you want to keep in mind. And VGP is a threat. Well, you could be a threat to the internet. So, you have to know what you're doing when dealing with VGP. So, who designed it? Whoever designed it thought that you have to be knowledgeable to run it. You can do a great deal of damage to the whole internet when you're running VGP. So, take for example the YouTube against Pakistan Telecom incident. Somebody in Pakistan started to announce the YouTube prefixes. And so, all the traffic directed to YouTube went to Pakistan instead of going to YouTube directly. And there are newbies playing around with MicroTik. Instead of doing, they created an ISPath of 256 entries. And they tickled the bug in all the other Cisco routers all over the world. And sessions all over the world were flopping around. And it took a while to figure that out for the whole internet. So, you know, you can't just email going back and forth to between people. Sorry? They were newbies in MicroTik. And didn't want to run, didn't want to read the manual. So, and so on. There are incidents like these pretty much every day. Some have a smaller influence on the internet, some have a bigger one. And yes, we showed it as possible. Now it's up to you. And one of the things I'd like to highlight is if you run, if you have a network, if you have an ISP, you want one of these probes to be in your network. Because there, this is a project from the RIPE NCC to monitor the internet. So, if you want one, just let me know. And I can ship you one from my office. So, you can install one of these probes at home in your office network. And you can monitor the, you can use them to monitor the whole internet from there. Or you can, you can use, you can use a, you can, you can play with one of these probes and get points using them to run your own test measurements against your own network. So, if you want to say, I have 500 probes all over the world, ping one of your IP addresses to see the reachability. You can do that from the, from the Atlas website. Or you can, you can run any, any other kind of measurement. So, it's, for now you can only do ping trace route and DNS lookup. But it's going to become a, it's a toolbox to run your own measurements on. World effort worldwide. Do you have any other questions? No questions? We're either. I just, looking at some of the technologies that you're using. And just from what I've read. Yeah. And it seems like a lot of the technologies you're using are some, not most, are the same that are integrated into ISP today, and like, correct, with the exception of somebody obviously, the content of the pre-infrastructure component. You mean for the, for the streaming part? Well, ISP config is more for, it's more suitable for hosting, hosting sites. So that, I didn't get into that detail because I was mostly focusing on the access part. But yes, some of, most of that is, well, I didn't get into the web server side of the, of the operations, but web server or email or whatever else side of the operations. But yes. What do you do with the accounting, the DNS? Yes, well, well, DNS, when you do access you want to just generally manage only reverse DNS for your customers. And again, I ended up writing my, my own software. It was actually integrated into the IP address management software where customers could just log in and see their IP addresses and set up the reverse DNS. It's trivial. Very, very, very trivial. So, yes, I didn't get into that detail because I was focusing more on the routing part. But yes, it's, it's mostly standard software. Yeah. So, yes, I'm, well, BGP is, why do they have that? So, yes. So, for the rest is pretty much common. Any other question? Do you build a time on the transit? Yeah. No, not the transit. A transit we do on antico-person-tile to, well, we have some co-location customers and we do on antico-person-tile on them. How do you build a time on that? MRTG. Sorry? Yeah. We're not very strict because actually, as, again, we go back to the content versus eyeballs, content and eyeballs. If you, if you keep a good balance between outgoing and incoming bandwidth, you don't want to care about what your customers are really doing. I mean, well, you still have to keep an eye on that, but you're not so strict. Unless you really want to charge them based on each and every megabit that they're doing, there's no reason to do that. So far, we do it just look at the graphs at the end of the month and that's it. Yeah. Yeah. Yeah. Yeah. Same thing. Unless, unless they're, they, they bought like 10 megabits of transit and then they do 20 or 25, well, then yes. They generally send them an email telling them, hey, this month you did this person pays more than you should be allowed. Are you guys comfortable with the 95th person tell how it works? No. Okay. So basically you run MRTG and you, and you, you get a sample of the traffic every five minutes. So you, at that point, you know how much traffic has been passing by on that, on that interface. And at the end of the month, you discard the highest 5% of those, those readings and then you get the average out of that. So you see that on, on average, your customer has done 20 megabits and you discard the 5% higher readings. So that means that he can do for 36 hours a month, he can do say if he bought 10 megabits of transit over 100 megabits port, up in port, they can do 100 megabits for 36 hours a month. Which means that if they have spikes or if they, if they want to do, I don't know, advertisement campaigns, they can still do them and not be built for them. And while you, while you might have big, big spikes on your network, you can still be sure that if they do more, if they use more traffic, then they will get built, or you can build it. Sorry? Exactly. Easy to put into this, you get the old, the old, the old 30 megabits. So. Quite a long, this may already be 30 hours. But what, what generally happens is that, I know some, some of the customers will pay 20 megabits and then the 95th person to be able to do 21 on average. You don't want to start billing them for 21. You just play the good, the good part and send them an email and say, hey, you know, you've been doing 21 megabits and then you might think about getting 25 of transit instead of just 20. And by the happens rarely because, well, either you are proactive or when you have both content and, and access on your, on the same network, you can, if you start having a lot of access, you can start actually giving content bandwidth away for free. You can start playing with it like using it as a commodity to encourage customers to stay with you. Sorry? Yeah, some costs because you're, you're always paid for, for bandwidth in both directions. So you will, if you're, if you're having access and you have 30 megabits of traffic for your access network, you will have 30 megabits getting to your network. And then you will still not, you might not still be sending too many bits out of your network. So if you, if you get co-location customers and you, and they will start sending bits out of your network, that's bandwidth you're already paying for. And then you can start playing with it from a sales perspective, sounding it for very cheap. Well, there's a case here in Ontario, Canada, there's a very large business, a very large business model, I mean, if anyone supported around code, many did second-hand code pump from 2007, Thank you.
that is different from the real world. Well, about the streaming services. So who are we? I am what is this about? The idea is to open up discussion for people to use open source software people to start using software solutions to talk, participate, and you can also set up an connection service, will bring you the access, goal was to get a balanced ratio between outgoing small and you have to start somewhere? So really way. So it's a matter of giving them a chance to vendors and or software. So, routing, how do you then do IPv4 on top of that. What are the routing solution where we are doing blackhauling. So this tunnel towards some of our customers because we we do, we look into the PPPOE session, get the what we have in the PPPOE table from this countries, then you want to start having things in the, in the software, but not yet. Do you might want to have XBGP running on the same with the audio. Now, a CDN is made out of deliver the bits as close as possible to the end a way, a technical way to make sure that the end you're doing really a network over the network, example, accounting information or content everything about all the content that needs to be This is a sort of big difference because we focus or things like that, and you want to use videos together, show things, popping up and down distributed to all the content distribution knowledgeable people. One of the challenges that you have a network, if you have an ISP, you want the technologies you're using are some, not most, only reverse DNS for your customers. And again, I and not be built for them. And while you, while already paying for. And then you can start