Introduction to Ceph cloud object storage
Wasserman, Orit
Ceph is a highly available distributed software defined storage, providingobject, key/value and file-system interfaces. Ceph Rados Gateway (Radosgw)provides HTTP REST API that is S3 and openstack swift compatible. This talkwill cover cloud object storage concepts and how Ceph implementation of cloudobject storage (Radosgw). This talk will also present the newest features andour plans for the future.
 This is Orit from the CoreSafe team. We can talk a little bit about our object storage, the RATOS Gateway in SEP. Hello. Thank you for coming so early on Sunday. I know it's hard. I'm Orit. I'm part of the SEP team in RATOS. And I work on Cloud of the Storage in SEP. And today we're going to explain what Cloud of the Storage is, how it's different to regular storage types, a little bit about SEP. So first, who uses SEP? Not much. We need more hands. So we explain what SEP is, how it's built, and then we talk about for this gateway, who is the part of SEP that provides Cloud of the Storage interface. And of course we have time for a few questions. So in general, I prefer people to ask questions during the presentation, so if someone needs a question, just raise your hand. So let's start. What is this Cloud of the Storage? So we have a block stage. Anybody who uses block stage? Yeah, of course. We all do. So the stage is divided into those fixed blocks depending on the device. We then have any metadata related to the data. You just write the data somewhere in the device of SEP left. And the application needs to manage to know what exactly it written there, where it's written. You can have several devices. But it's really fast. Well, first of all, it. And today we're going to say it's really fast. We have the other protocol, fibercharm, SCSI, SATA, iSCSI, and some people even use fibercharm over internet. But using block storage really hard for applications. So then, can file systems. So here the data is organized. We have higher fee of directories that contain files and other directories. We have users. We have metadata for this data. The file can be in any place. You can override it in the middle and allocate on demand. It's much easier for the application to use the storage. But it's not in block. That's a argument. Sometimes five system people, but it's slower. Because of the all metadata and you are here on the link. You can server not all files systems, but many have shown semantics. So you actually can log the file for read only or completely for you. And two different applications can use and write to the same file at the same time. We have local file system protocol like XC4, XFS and so on. And we also have network file protocol like NFS and SMB. And F2 is the Apple protocol by the class they decided not to use it anymore. And they're moving to SMB. But then came something new. It was the cloud. So we had that problem before the cloud came. We saw the story is going and going. So people moved to more distributed storage systems. And writing a distributed file system is really hard. I don't know if many of you have tried writing a regular file system hard, but when you try to distribute it, the metadata, you need to think it all around, it's a problem and there are a problem. The cluster people are you can say. But in the cloud it became even harder because we're not only talking about a large amount of data that it accessed all around, but we're talking also a large, really big data chunks. And then came the object storage that existed before the cloud object storage. But it was mostly like a niche, I guess, and I did research stuff on object storage, but it wasn't user-doer-standard. But nobody used it. But then came Amazon, actually. And when the site was on, object storage collapsed. I remember it was something like, I guess, ten years ago when S3 was launched, and I thought, wow, this is the way to do storage. How can we do that? I thought it really was amazing as a storage person. I was doing my research and actually said, no, we need to do something like writing an IBM. But of course, IBM is not good at being innovated and they never did anything like that. So the idea is very simple. One, we're talking about cloud environment. So you want the API to be really for cloud. So that means it should be HTTP-based, so that's West API. And you learn some of the amazing of the data, but the complex IRHF-like systems have is really hard. So it's just two levels. Amazon formed them buckets, Swift called them containers, pools. So just the container of objects is the way to organize your object into groups. And maybe add some properties for that. So that's bucket. And inside the bucket, you have those objects. An object are not just data. You have a name, you can add them all sort of metadata. Some are fixed and some you can decide on your own. So you can let them find them and know what's the data in them. You have users and tenants, which is like an umbrella for user to share the same data between them. You have authentication and ownership, access control list. And you talk about large objects, really large. So that means you need lots of storage. Because there's a large object, usually objects are immutable. That means you write them once, if you decide to change them, you need to actually override all the objects. You cannot write in between. And it's much more efficient for those large objects to be immutable and it simplifies the allocation. So the protocol today, the most common one is Amazon S3. It's the richest one. And I don't know what scale Amazon is, but I'm sure it's huge. I cannot imagine how much storage S3 has. In AppStack we have Swift and there's also Google Cloud Storage. So this is the ground around the API links. Put writes an object, get... This is a bucket. So put creates an object, bucket, get with the bucket. You have actually... it can be more complex because you can read also the data that are using head. You can delete a bucket, create an object. But it's not a simple copy of version. This is a simplification because as you see there's some headers there. Usually there's lots of header that says all sorts of metadata or maybe make the operation more complex. So we talked about really large objects. Anyone here has a downloaded stuff from the internet and you lost connection and you need to download. Everybody. It's really annoying. And then we talked about larger books. We can get to jiggle size. This is can be really hard. So for that we have the multi part. You want to upload a download an object but you want to maybe stop it, maybe we try. So we take a big object, you divide it to small parts and each part is being uploaded and downloaded parallel so you can handle network problems. And then you can continue where you stopped. And you can stop start, upload download. And also it's good for streaming for example. When you don't know the size of the objects you're actually going to upload or download. Download usually knows but upload sometimes generate the data when you upload it and for that you use multi part. But it's harder for the storage because until you finish your upload you don't know what the object will be and we cannot commit it. So we need to start all those temporary parts. And sometimes you have failure and we start the upload completely again. So we need to in the background to clean all the extra stuff we got. Another feature which I really like. So here we have used here VMS. So remember VMS, we have the file versioning. When you write on the file you get a dot something. I really like it because it sometimes do by mistake the leaf stuff. It will be good, you always have the previous copy. So again in cloud storage we have versioning, where we fold buckets, it's per bucket. But if all buckets are not with versioning but you can enable it, that's mean. When you override the object it creates a new object with a different version. If you delete the latest you will still have the old version. It's really helpful when you delete stuff by mistake or overwrite by mistake an object. And cloud objects are really cool stuff. I didn't talk to you about authentication because every protocol does it completely different. The main difference between Google Cloud Storage and S3 is authentication and it's incompatible. So it's different. Some use user passwords, some use user and a key and a secret. Everything is really different. So authentication is complex and we have object lifecycle. For example you can set a date on the object and when it gets to that date, called object exploration, that's what can be deleted automatically or moved to cloud storage. And lots of cool features. Any question about cloud storage? So let's talk about S3. So S3 feels so close to the circle pod which is the family of bucket poses in school. So all our versions are called one animal that name. So the latest stable version is dual. So self is open source. This is our give up. We have set all under it. It's all the self code. It's not only self. We have our own testing framework called totality. It's also open. And we have additional tests like we have S3 tests. There's S3 compatibility. We have other components of self. So self is open source. We are in problem. So it's open source, LGPL. It's software defined storage. Well, we are in the software defined storage room. That's mean it can run on many kind of platforms. You can configure and adapt it to different apologies and scale. It's a bit system and like you say, it doesn't have any single point of failure. When you have a large number of nodes, you cannot handle a single point of failure. Because you assume everything will fail. Large scale means failure happens. It's massively scalable. It was built to be large scale. So we serve PLC and build three nodes, but the minimum is five. And I actually recommend more. It's a large scale system, not for two nodes, three nodes. Because we are large scale, we assume that we have to either replicate data or use the words according to be more efficient space. Because we assume they don't have much left, so we need more copies. It's a large system. Failures happen. So we want self-healing because you cannot handle those failures manually. Everything should happen automatically. And we have unified access. So we have self-healing. It's the file system access. It's a positive client file system. You can use our client fields or our kernel client. And there's integration to Opus.Memilar. And you can use NFS Ganesha with it for NFS and Samba for SMB. And it is the block interface. Many know it from Opus.Tap. It's integrated completely with QMU. So you can use it with QMU for Xen or QMU for KVM. And you can use the kernel client if you want your own look device. And we have BodeSketway, which is a component that provides cloud object storage. Even if this all were WebOS, and I'm going to speak bit about WebOS. WebOS is a reliable, autonomous, distributed object storage. So as you see, it's an object storage, which is interesting. So when self-started, it was to build a very fast, scalable, distributed file system. And the style would rather us to be as the rich object storage that you can build the file system on it. Easier, please. But I can distribute a file system. It's never simple, even if you have distributed objects storage underneath. And in the meantime, we took to develop CFFS, the style of IBD, and WebOS Gaitway. And now we also have CFFS as product. So what this does, it does all the distributed stuff. So it does all the replication and all the ways you're coding. It's a flight object name-stance, and we call, we have a pool of objects. And you can configure it to pool. It could be a pool of very fast storage, or pool of slower storage with high-risk. It could be different applications. It could be freeways, or it could be res-accords. CFFS is a strong ecosystem system. And it's software-defined, that means it's aware of the infrastructure and the topology, and it matches two different architectures and systems. For placement, we have a HustBase algorithm with no lookup called Crush. And we want performance, so we serve the data directory to the client. So we wrote about Crush. Crush is our placement algorithm, and we wanted an algorithm because it's a large-scale system that doesn't require any lookup. Lookup requires some central area. You keep the table of the lookup. You can replicate it, but it's not very efficient in large scale. It's a bottleneck. So Crush doesn't require any lookup. It is topology aware, so that means you can actually put in cloud. You can say, this is work, and I don't want to place two copies on the same work. So you choose nodes that are not on the same work. You can configure the application. You can do weights. And it's really fast-capsulation, deterministic, and it's evenly distributed. That's why we call it Soderander because that's where we can benefit from all nodes. So a surf wireless cluster has two kinds of nodes. The first one is the object storage device. Those we have lots of nodes, demons. Tens to ten harvests. I think ten harvests is a lot, maybe thousands. And we have one OSB per disk. You can put an OSB on a wave, but it's a waste because we do the application ourselves. There's no reason to do additional replication in the storage layer. And its main thing we do is set the object to the user. It's a smart storage node that's meaning that's peering. And in case of a failure, it does the recovery automatically. There are always these talking with each other and we'll work with KDN. If a new node comes, it will rebalance the storage. The archive of nodes we have is the monitor nodes. They store all the clustering logic. They maintain the cluster membership. So if a node fails, the monitors get a notification because it still notifies that the OSB is not responding. And it will notify the monitor there's a failure, and then the monitor will spread it to the other OSBs. And when you add an OSB in the same way, we have a small number of those nodes, three, five, so on. They do a list of boxes. It doesn't stop any data to the user. It's just for the cluster membership data. So we have this cool object storage cluster. We want to accelerate. So we do need webis. And you actually can use webis if you have your own application. And it really needs lots of performance and you're willing to interact with the webis API directly. And you can use webis. Actually, some users will use webis directly. It has been into C++, a web will be Python. I don't remember. Probably more. And it's a very rich API. It's not just a sweet, wide object. You have the key value store. We call it OMAP. So inside one object, you have a key value store. And you can use this to store. For example, the five system stores. The director is in an OMAP and we'll just get where the bucket index is OMAP. It's a single object operation so you can update attributes and keys and data in one operation. It has snapshots for objects. Objects are mutable. That means you can actually partially override objects. And webis clusters are very similar to storage procedures in a database. You can actually write code that will run directly on the OSD when an object is changed. It's really a strong feature. It allows lots of performance. We use it all around the services. And we have a watch notify service so we can register an object. And when the object changes, we get a notification across the cluster. Everyone registers notification. So it's a very strong API, but it's not a standard API. It's not block interface, file, object, cloud object. It's your one. And if you really import a performance and you really need to make the effort, you'll be really cool. Any questions about self in German? Just a few words about self in English. Okay. So first, all these are divided into placement groups. And as we all have, we need to replicate one another. One can be the buster in some cases. This other case is just a replica. So they talk with each other and they can detect when there's failure. Because they need to talk with each other and then if they get in error, they notify the monitor. And then the cluster starts to under that case. So you need that FD that failed. It's taken while to mark it as failed. But when it's failed, you know all the data it contains. And it will be replicated to other SDs. So then they do it automatically. One more question about each object. Each node has a complete set of the whole object. Well, depending on the size of the object. But our maximum size of object in the radios is 4M. So 4M will be, let's say you have freeways replicates. So freeways will have those 4M. If the object is larger, then it's more complete. My question is, is the object distributed? That it means you have to have a minimum set of storage nodes to have to retrieve the complete object. No. To synthesize? No. You need just one. Yeah, you need one to read. To write, you need three. If it's freeways, you need three copies. But if you read, you need one copy. Yes? Is it better for set many lower quality machines, or is it better to have fewer number, but higher quality machines? Many lower. If that's the demand, of course we prefer many high quality. But the more you have machines, the more you can do large scale. Now we're going to talk about more about FEDIS Gateway, which is the component in FEDIS that provides the cloud of the search. So all the components of FEDIS Gateway uses Li-Ve-DOS to communicate with the radios clusters. So we have these clusters, voids, ds, and monitors, and that's where the radar is stored. And this gateway is a service built on top of that. So wireless is object storage. So that's good. It should be simple. We can have cloud object storage. But life is never simple. So it's not just to provide the rest of the API and matching the API we have. When we look at our object for radios and other security way, or cloud, we see some difference. So first radios, the biggest size of an object is four mega, and we're talking about cloud, so we talk about three really large objects. So we need some way of taking bigger objects and dividing them to those four mega-size objects. So radios objects are mutable, but in cloud we talk about immutable objects. The hardest part is that the poles inside of radios are not indexed, but user want to list all the objects in the bucket sorted by name. That means we need to add some indexing for the bucket to allow those listing. And radios as per pole are CL. So pole is like a bucket, but we need a CL as per object. So even in spiral-structured radios, is object, we still need to add layer to implement the cloud of the storage at the top of it. So we have the radios cluster. We save all the data inside the radios cluster. Radios gateway is stateless, so in case you see that you have lots of actions and operations, and the radios gateway is loaded, you can just run another instance of radios gateway on the same set cluster and both can work together and continue. And that way you can scale up. So again this is object storage. We have users and tenants. We have buckets, objects. We have metadata, per bucket, per object. We have access control list, per bucket, per object. We have authentication, which is really complex because we support server protocols. Today we support two-row cloud of Rixx protocol, S3 and Swift. In many cases you can use both in the same time, but sadly they are not compatible, so there are cases that it won't work. If you can't upload multi-part with Swift and with it, we'll mess with it because they sign differently the calculator, the signature differently. Versioning is very different. You cannot use both. Version bucket for S3 would not be versioned when you access this from Swift and other way. The authentication of course is very different, but we do support Keystone for S3, so that's maybe okay. We also support NFS. And it says that NFS is a file protocol. Yes, it is. It's not a full NFS. If your main work order is a file system, the CFFS, it's for users that mostly use object storage and they can use the NFS. We use NFS Gimisha on top of a library called libhgw to allow that. And it's for migrating from NFS to object storage or exporting a bit. And also if you have one, this legacy application that made this NFS very basic stuff, then you can use it. But for full-scale NFS, you use FFS or other NFS stuff. So we try to build the latest gateway as very accessible. So a request goes for all the layer one after the other. We have two layer, three parts that everyone can access all the time. I'll go a bit about those. So first we have the frontend. Frontend is what provides the rest API actually. So it has to be HTTP. So with two layers, the first is the old and un-recommended way. So we support fast-synth VGI with Apache and probably other web servers. The reason it's not recommended is because fast-synth GI is lots of security issues. But sometimes if you already have your own Apache and you handle the security issue with fast-synth GI, it can be an option. Today we have also, from Hama, five-flat actually, I don't remember. We have Sivot Web. It's in a very, very server-inside of the web disk gateway. So we recommend that. Then we go to the rest layer. It's actually the layer that converts from past the dialect between the protocol. So it passes S3, S3, S3, then RAPI. Then it goes to execution layer, which is the common layer for protocols. We don't run to IDECOR twice. So we try to use as much as common code, handling different protocols. The next layer is the layer that talks to Redis. It's not enough to use Redis. We need to do some stuff on our own. So for example, we do object striping, atomic overrides, and all the bucket index handling. And we also, it contains the object classes, the run and the SD. We have quarter, and we have quarter per user, quarter per bucket. We have the execution layer because in many cases, in many layers, we need to attend, sometimes we need to educate again. So we support AWS-free, AWS-free, Keystone-2, Keystone-B-3, LDAP. This is what we're going on on this US, but hopefully it will be in the end of CoC or maybe in the winners. Objects are large, so we don't delete them and free the space immediately because it takes time. So it will move to a garbage collection in the background as a process that contains the large objects. I don't have time, so I'm going to skip some slides. So this is our rebuild object. We take an object with a head object and pull. All objects have a head object. It contains all the object metadata it attributes. Up to 512 cases of data. Small objects only have an head. Large objects are stripes and several tail objects. To have a fast access to objects, objects are names inside of this. So they have the number 123, the bucket ID. It's not the name because bucket can be named bucket, so it's a unique ID for bucket. And then the object name. That means that we don't need to do any lookup in order to read the head because it contains the metadata and we want many cases to be able to do it fast. And then the tail is the bucket ID and then some UUID and the pot. A little bit of bucket index. So bucket index is just all the objects in the bucket sorted by name. But when we talk about clouded storage with cases that one bucket contains millions of objects. And then the bucket index is also a bottleneck for performance and also because it's in the end stored in an older weather subject. If we pass one object line, it's also inefficient. So first thing, if you have lots of larger objects, the search number is up to whatever K object. One bucket in this, that's fine. If you have more than one under case, then you need to use shouting. We collect the bucket index and split it to several objects. Depending on the number of objects, you decide the number of shards you want. We have now support for offline reshouting. So in case you didn't reshout the bucket and suddenly you got to live number of objects, you can reshout it. I think it's in the summer, the summer, the 10, and it's in the new versions. And we're trying to work on the online reshouting that you will need to offline the bucket and stop pirating it. You can do it with Ion. We skip that. So our quota is the redistributed system. Objects can be written from many, many places, several radios get ways. We need some way to coordinate the quota. So we use the radiosubject class objects to do that. So when we actually write the objects in the RSD, we update the quota. That means the quota is not completely consistent with sometimes a myth a bit. But I think five systems have the same issue. We have the metadata cache and we use watch notify to handle that. That's mainly in case the metadata change, the other way is get an application and then can invalidate that entry from the cache. So multi-site. So we're talking about cloud, but cloud is never one data center. Many times we want several data centers and we want to have higher disaster recovery. We actually want to use all of them. For that we have multi-site, our application. So you can take two-step clusters and you can configure them. And there will be a sequence application on one of each other. It can be active, active or passive active. The metadata operation, which is use the equation, use the solution, bucket question, bucket delivery, and a few bucket metadata operations, are sequenced because they are really important stuff and we cannot have a difference in those. So we have a meta-master and in case he fails, you need to do failover and configure a new meta-master. But the data is completely asynchronous. That means all different clusters, you cannot build the object and there will be replicated automattically to the allocasters. A zone is what we call a self-cluster. So if you have two locations, each one will have a self-cluster that will be two zones. A zone group is just a group of zones that share the same data. So even not in the same location, they will be in the same zone group because they will be replicated from each other. We have all multi-site support in HANA. It's called a sync agent. It's only active passive. But from June, everything is active active. I don't have time. Okay. More stuff. So we talk about object life cycle. We don't support the full S3 object life cycle. We have object operation and basic storage policy, which means that after the object expires, we move it to a different called storage. We support object. We add more efficient object called the operation. We don't actually copy the data, but just update the object head to point. So now it points to the same data as a different object. We have a question. Thanks to me. So we object to our trip. We have compassion. Also, we want to work with support. We have static website. Anyone wants to use that website. We have support. That's new in Khafen. It's not active. We have met a little search. We can do exporter data to elastic search and that's where the user can search the metadata. And we're working on doing online backlit resharing. I hope I have time to questions. No. Okay. Questions. One. Yes. Maybe what is the performance parameter for the rather state-based data? Well, you ask what's the performance parameter of using rather state-based data and not rather state-based data. So we need to understand that we are very different. Rather, this is REST API. So REST API is just a PPP. That's low. It's a slow protocol like PPP. So I think that's the main performance parameter. But if you're talking about really, really large objects, then I think the REST API is limited. We have some of the REST. We are not as large as Bench because we have per-object also. We have a few new objects, not just an object. So I don't know the numbers, but those are currently. Thank you.
 This is Orit from the CoreSafe team.  We can talk a little bit about our object storage, the RATOS Gateway in SEP.  Hello. Thank you for coming so early on Sunday. I know it's hard.  I'm Orit. I'm part of the SEP team in RATOS.  And I work on Cloud of the Storage in SEP.  And today we're going to explain what Cloud of the Storage is,  how it's different to regular storage types, a little bit about SEP.  So first, who uses SEP?  Not much. We need more hands.  So we explain what SEP is, how it's built, and then we talk about for this gateway,  who is the part of SEP that provides Cloud of the Storage interface.  And of course we have time for a few questions.  So in general, I prefer people to ask questions during the presentation,  so if someone needs a question, just raise your hand.  So let's start. What is this Cloud of the Storage?  So we have a block stage.  Anybody who uses block stage?  Yeah, of course. We all do.  So the stage is divided into those fixed blocks depending on the device.  We then have any metadata related to the data.  You just write the data somewhere in the device of SEP left.  And the application needs to manage to know what exactly it written there,  where it's written. You can have several devices.  But it's really fast.  Well, first of all, it. And today we're going to say it's really fast.  We have the other protocol, fibercharm, SCSI, SATA, iSCSI,  and some people even use fibercharm over internet.  But using block storage really hard for applications.  So then, can file systems.  So here the data is organized.  We have higher fee of directories that contain files and other directories.  We have users. We have metadata for this data.  The file can be in any place.  You can override it in the middle and allocate on demand.  It's much easier for the application to use the storage.  But it's not in block.  That's a argument. Sometimes five system people, but it's slower.  Because of the all metadata and you are here on the link.  You can server not all files systems, but many have shown semantics.  So you actually can log the file for read only or completely for you.  And two different applications can use and write to the same file at the same time.  We have local file system protocol like XC4, XFS and so on.  And we also have network file protocol like NFS and SMB.  And F2 is the Apple protocol by the class they decided not to use it anymore.  And they're moving to SMB.  But then came something new.  It was the cloud.  So we had that problem before the cloud came.  We saw the story is going and going.  So people moved to more distributed storage systems.  And writing a distributed file system is really hard.  I don't know if many of you have tried writing a regular file system hard,  but when you try to distribute it, the metadata, you need to think it all around,  it's a problem and there are a problem.  The cluster people are you can say.  But in the cloud it became even harder because we're not only talking about a large amount of data  that it accessed all around, but we're talking also a large, really big data chunks.  And then came the object storage that existed before the cloud object storage.  But it was mostly like a niche, I guess, and I did research stuff on object storage,  but it wasn't user-doer-standard.  But nobody used it.  But then came Amazon, actually.  And when the site was on, object storage collapsed.  I remember it was something like, I guess, ten years ago when S3 was launched,  and I thought, wow, this is the way to do storage. How can we do that?  I thought it really was amazing as a storage person.  I was doing my research and actually said,  no, we need to do something like writing an IBM.  But of course, IBM is not good at being innovated and they never did anything like that.  So the idea is very simple.  One, we're talking about cloud environment.  So you want the API to be really for cloud.  So that means it should be HTTP-based, so that's West API.  And you learn some of the amazing of the data,  but the complex IRHF-like systems have is really hard.  So it's just two levels.  Amazon formed them buckets, Swift called them containers, pools.  So just the container of objects is the way to organize your object into groups.  And maybe add some properties for that.  So that's bucket.  And inside the bucket, you have those objects.  An object are not just data.  You have a name, you can add them all sort of metadata.  Some are fixed and some you can decide on your own.  So you can let them find them and know what's the data in them.  You have users and tenants, which is like an umbrella for user to share the same data between them.  You have authentication and ownership, access control list.  And you talk about large objects, really large.  So that means you need lots of storage.  Because there's a large object, usually objects are immutable.  That means you write them once, if you decide to change them,  you need to actually override all the objects.  You cannot write in between.  And it's much more efficient for those large objects to be immutable and it simplifies the allocation.  So the protocol today, the most common one is Amazon S3.  It's the richest one.  And I don't know what scale Amazon is, but I'm sure it's huge.  I cannot imagine how much storage S3 has.  In AppStack we have Swift and there's also Google Cloud Storage.  So this is the ground around the API links.  Put writes an object, get...  This is a bucket.  So put creates an object, bucket, get with the bucket.  You have actually... it can be more complex because you can read also the data that are using head.  You can delete a bucket, create an object.  But it's not a simple copy of version.  This is a simplification because as you see there's some headers there.  Usually there's lots of header that says all sorts of metadata  or maybe make the operation more complex.  So we talked about really large objects.  Anyone here has a downloaded stuff from the internet and you lost connection  and you need to download.  Everybody.  It's really annoying.  And then we talked about larger books.  We can get to jiggle size.  This is can be really hard.  So for that we have the multi part.  You want to upload a download an object but you want to maybe stop it, maybe we try.  So we take a big object, you divide it to small parts  and each part is being uploaded and downloaded parallel  so you can handle network problems.  And then you can continue where you stopped.  And you can stop start, upload download.  And also it's good for streaming for example.  When you don't know the size of the objects you're actually going to upload or download.  Download usually knows but upload sometimes generate the data when you upload it  and for that you use multi part.  But it's harder for the storage because until you finish your upload  you don't know what the object will be and we cannot commit it.  So we need to start all those temporary parts.  And sometimes you have failure and we start the upload completely again.  So we need to in the background to clean all the extra stuff we got.  Another feature which I really like.  So here we have used here VMS.  So remember VMS, we have the file versioning.  When you write on the file you get a dot something.  I really like it because it sometimes do by mistake the leaf stuff.  It will be good, you always have the previous copy.  So again in cloud storage we have versioning, where we fold buckets, it's per bucket.  But if all buckets are not with versioning but you can enable it, that's mean.  When you override the object it creates a new object with a different version.  If you delete the latest you will still have the old version.  It's really helpful when you delete stuff by mistake or overwrite by mistake an object.  And cloud objects are really cool stuff.  I didn't talk to you about authentication because every protocol does it completely different.  The main difference between Google Cloud Storage and S3 is authentication and it's incompatible.  So it's different.  Some use user passwords, some use user and a key and a secret.  Everything is really different.  So authentication is complex and we have object lifecycle.  For example you can set a date on the object and when it gets to that date, called object exploration,  that's what can be deleted automatically or moved to cloud storage.  And lots of cool features.  Any question about cloud storage?  So let's talk about S3.  So S3 feels so close to the circle pod which is the family of bucket poses in school.  So all our versions are called one animal that name.  So the latest stable version is dual.  So self is open source.  This is our give up.  We have set all under it.  It's all the self code.  It's not only self.  We have our own testing framework called totality.  It's also open.  And we have additional tests like we have S3 tests.  There's S3 compatibility.  We have other components of self.  So self is open source.  We are in problem.  So it's open source, LGPL.  It's software defined storage.  Well, we are in the software defined storage room.  That's mean it can run on many kind of platforms.  You can configure and adapt it to different apologies and scale.  It's a bit system and like you say, it doesn't have any single point of failure.  When you have a large number of nodes, you cannot handle a single point of failure.  Because you assume everything will fail.  Large scale means failure happens.  It's massively scalable.  It was built to be large scale.  So we serve PLC and build three nodes, but the minimum is five.  And I actually recommend more.  It's a large scale system, not for two nodes, three nodes.  Because we are large scale, we assume that we have to either replicate data or use the words according to be more efficient space.  Because we assume they don't have much left, so we need more copies.  It's a large system.  Failures happen.  So we want self-healing because you cannot handle those failures manually.  Everything should happen automatically.  And we have unified access.  So we have self-healing.  It's the file system access.  It's a positive client file system.  You can use our client fields or our kernel client.  And there's integration to Opus.Memilar.  And you can use NFS Ganesha with it for NFS and Samba for SMB.  And it is the block interface.  Many know it from Opus.Tap.  It's integrated completely with QMU.  So you can use it with QMU for Xen or QMU for KVM.  And you can use the kernel client if you want your own look device.  And we have BodeSketway, which is a component that provides cloud object storage.  Even if this all were WebOS, and I'm going to speak bit about WebOS.  WebOS is a reliable, autonomous, distributed object storage.  So as you see, it's an object storage, which is interesting.  So when self-started, it was to build a very fast, scalable, distributed file system.  And the style would rather us to be as the rich object storage  that you can build the file system on it.  Easier, please.  But I can distribute a file system.  It's never simple, even if you have distributed objects storage underneath.  And in the meantime, we took to develop CFFS, the style of IBD, and WebOS Gaitway.  And now we also have CFFS as product.  So what this does, it does all the distributed stuff.  So it does all the replication and all the ways you're coding.  It's a flight object name-stance, and we call, we have a pool of objects.  And you can configure it to pool. It could be a pool of very fast storage,  or pool of slower storage with high-risk.  It could be different applications. It could be freeways, or it could be res-accords.  CFFS is a strong ecosystem system.  And it's software-defined, that means it's aware of the infrastructure and the topology,  and it matches two different architectures and systems.  For placement, we have a HustBase algorithm with no lookup called Crush.  And we want performance, so we serve the data directory to the client.  So we wrote about Crush.  Crush is our placement algorithm, and we wanted an algorithm because it's a large-scale system  that doesn't require any lookup.  Lookup requires some central area. You keep the table of the lookup.  You can replicate it, but it's not very efficient in large scale. It's a bottleneck.  So Crush doesn't require any lookup.  It is topology aware, so that means you can actually put in cloud.  You can say, this is work, and I don't want to place two copies on the same work.  So you choose nodes that are not on the same work.  You can configure the application. You can do weights.  And it's really fast-capsulation, deterministic, and it's evenly distributed.  That's why we call it Soderander because that's where we can benefit from all nodes.  So a surf wireless cluster has two kinds of nodes.  The first one is the object storage device. Those we have lots of nodes, demons.  Tens to ten harvests. I think ten harvests is a lot, maybe thousands.  And we have one OSB per disk. You can put an OSB on a wave,  but it's a waste because we do the application ourselves.  There's no reason to do additional replication in the storage layer.  And its main thing we do is set the object to the user.  It's a smart storage node that's meaning that's peering.  And in case of a failure, it does the recovery automatically.  There are always these talking with each other and we'll work with KDN.  If a new node comes, it will rebalance the storage.  The archive of nodes we have is the monitor nodes.  They store all the clustering logic. They maintain the cluster membership.  So if a node fails, the monitors get a notification because it still notifies that the OSB is not responding.  And it will notify the monitor there's a failure, and then the monitor will spread it to the other OSBs.  And when you add an OSB in the same way, we have a small number of those nodes,  three, five, so on. They do a list of boxes.  It doesn't stop any data to the user. It's just for the cluster membership data.  So we have this cool object storage cluster.  We want to accelerate. So we do need webis.  And you actually can use webis if you have your own application.  And it really needs lots of performance and you're willing to interact with the webis API directly.  And you can use webis. Actually, some users will use webis directly.  It has been into C++, a web will be Python. I don't remember. Probably more.  And it's a very rich API. It's not just a sweet, wide object.  You have the key value store. We call it OMAP.  So inside one object, you have a key value store.  And you can use this to store. For example, the five system stores.  The director is in an OMAP and we'll just get where the bucket index is OMAP.  It's a single object operation so you can update attributes and keys and data in one operation.  It has snapshots for objects.  Objects are mutable. That means you can actually partially override objects.  And webis clusters are very similar to storage procedures in a database.  You can actually write code that will run directly on the OSD when an object is changed.  It's really a strong feature. It allows lots of performance.  We use it all around the services.  And we have a watch notify service so we can register an object.  And when the object changes, we get a notification across the cluster.  Everyone registers notification.  So it's a very strong API, but it's not a standard API.  It's not block interface, file, object, cloud object.  It's your one. And if you really import a performance and you really need to make the effort,  you'll be really cool.  Any questions about self in German?  Just a few words about self in English.  Okay. So first, all these are divided into placement groups.  And as we all have, we need to replicate one another. One can be the buster in some cases.  This other case is just a replica.  So they talk with each other and they can detect when there's failure.  Because they need to talk with each other and then if they get in error, they notify the monitor.  And then the cluster starts to under that case.  So you need that FD that failed.  It's taken while to mark it as failed.  But when it's failed, you know all the data it contains.  And it will be replicated to other SDs.  So then they do it automatically.  One more question about each object.  Each node has a complete set of the whole object.  Well, depending on the size of the object.  But our maximum size of object in the radios is 4M.  So 4M will be, let's say you have freeways replicates.  So freeways will have those 4M.  If the object is larger, then it's more complete.  My question is, is the object distributed?  That it means you have to have a minimum set of storage nodes to have to retrieve the complete object.  No.  To synthesize?  No.  You need just one.  Yeah, you need one to read.  To write, you need three.  If it's freeways, you need three copies.  But if you read, you need one copy.  Yes?  Is it better for set many lower quality machines, or is it better to have fewer number, but higher quality machines?  Many lower.  If that's the demand, of course we prefer many high quality.  But the more you have machines, the more you can do large scale.  Now we're going to talk about more about FEDIS Gateway, which is the component in FEDIS that provides the cloud of the search.  So all the components of FEDIS Gateway uses Li-Ve-DOS to communicate with the radios clusters.  So we have these clusters, voids, ds, and monitors, and that's where the radar is stored.  And this gateway is a service built on top of that.  So wireless is object storage.  So that's good.  It should be simple.  We can have cloud object storage.  But life is never simple.  So it's not just to provide the rest of the API and matching the API we have.  When we look at our object for radios and other security way, or cloud, we see some difference.  So first radios, the biggest size of an object is four mega, and we're talking about cloud, so we talk about three really large objects.  So we need some way of taking bigger objects and dividing them to those four mega-size objects.  So radios objects are mutable, but in cloud we talk about immutable objects.  The hardest part is that the poles inside of radios are not indexed, but user want to list all the objects in the bucket sorted by name.  That means we need to add some indexing for the bucket to allow those listing.  And radios as per pole are CL.  So pole is like a bucket, but we need a CL as per object.  So even in spiral-structured radios, is object, we still need to add layer to implement the cloud of the storage at the top of it.  So we have the radios cluster.  We save all the data inside the radios cluster.  Radios gateway is stateless, so in case you see that you have lots of actions and operations, and the radios gateway is loaded,  you can just run another instance of radios gateway on the same set cluster and both can work together and continue.  And that way you can scale up.  So again this is object storage.  We have users and tenants.  We have buckets, objects.  We have metadata, per bucket, per object.  We have access control list, per bucket, per object.  We have authentication, which is really complex because we support server protocols.  Today we support two-row cloud of Rixx protocol, S3 and Swift.  In many cases you can use both in the same time, but sadly they are not compatible, so there are cases that it won't work.  If you can't upload multi-part with Swift and with it, we'll mess with it because they sign differently the calculator, the signature differently.  Versioning is very different.  You cannot use both.  Version bucket for S3 would not be versioned when you access this from Swift and other way.  The authentication of course is very different, but we do support Keystone for S3, so that's maybe okay.  We also support NFS.  And it says that NFS is a file protocol.  Yes, it is.  It's not a full NFS.  If your main work order is a file system, the CFFS, it's for users that mostly use object storage and they can use the NFS.  We use NFS Gimisha on top of a library called libhgw to allow that.  And it's for migrating from NFS to object storage or exporting a bit.  And also if you have one, this legacy application that made this NFS very basic stuff, then you can use it.  But for full-scale NFS, you use FFS or other NFS stuff.  So we try to build the latest gateway as very accessible.  So a request goes for all the layer one after the other.  We have two layer, three parts that everyone can access all the time.  I'll go a bit about those.  So first we have the frontend.  Frontend is what provides the rest API actually.  So it has to be HTTP.  So with two layers, the first is the old and un-recommended way.  So we support fast-synth VGI with Apache and probably other web servers.  The reason it's not recommended is because fast-synth GI is lots of security issues.  But sometimes if you already have your own Apache and you handle the security issue with fast-synth GI, it can be an option.  Today we have also, from Hama, five-flat actually, I don't remember.  We have Sivot Web.  It's in a very, very server-inside of the web disk gateway.  So we recommend that.  Then we go to the rest layer. It's actually the layer that converts from past the dialect between the protocol.  So it passes S3, S3, S3, then RAPI.  Then it goes to execution layer, which is the common layer for protocols.  We don't run to IDECOR twice.  So we try to use as much as common code, handling different protocols.  The next layer is the layer that talks to Redis.  It's not enough to use Redis.  We need to do some stuff on our own.  So for example, we do object striping, atomic overrides, and all the bucket index handling.  And we also, it contains the object classes, the run and the SD.  We have quarter, and we have quarter per user, quarter per bucket.  We have the execution layer because in many cases, in many layers, we need to attend, sometimes we need to educate again.  So we support AWS-free, AWS-free, Keystone-2, Keystone-B-3, LDAP.  This is what we're going on on this US, but hopefully it will be in the end of CoC or maybe in the winners.  Objects are large, so we don't delete them and free the space immediately because it takes time.  So it will move to a garbage collection in the background as a process that contains the large objects.  I don't have time, so I'm going to skip some slides.  So this is our rebuild object.  We take an object with a head object and pull.  All objects have a head object.  It contains all the object metadata it attributes.  Up to 512 cases of data.  Small objects only have an head.  Large objects are stripes and several tail objects.  To have a fast access to objects, objects are names inside of this.  So they have the number 123, the bucket ID.  It's not the name because bucket can be named bucket, so it's a unique ID for bucket.  And then the object name.  That means that we don't need to do any lookup in order to read the head because it contains the metadata and we want many cases to be able to do it fast.  And then the tail is the bucket ID and then some UUID and the pot.  A little bit of bucket index.  So bucket index is just all the objects in the bucket sorted by name.  But when we talk about clouded storage with cases that one bucket contains millions of objects.  And then the bucket index is also a bottleneck for performance and also because it's in the end stored in an older weather subject.  If we pass one object line, it's also inefficient.  So first thing, if you have lots of larger objects, the search number is up to whatever K object.  One bucket in this, that's fine.  If you have more than one under case, then you need to use shouting.  We collect the bucket index and split it to several objects.  Depending on the number of objects, you decide the number of shards you want.  We have now support for offline reshouting.  So in case you didn't reshout the bucket and suddenly you got to live number of objects, you can reshout it.  I think it's in the summer, the summer, the 10, and it's in the new versions.  And we're trying to work on the online reshouting that you will need to offline the bucket and stop pirating it.  You can do it with Ion.  We skip that.  So our quota is the redistributed system.  Objects can be written from many, many places, several radios get ways.  We need some way to coordinate the quota.  So we use the radiosubject class objects to do that.  So when we actually write the objects in the RSD, we update the quota.  That means the quota is not completely consistent with sometimes a myth a bit.  But I think five systems have the same issue.  We have the metadata cache and we use watch notify to handle that.  That's mainly in case the metadata change, the other way is get an application and then can invalidate that entry from the cache.  So multi-site.  So we're talking about cloud, but cloud is never one data center.  Many times we want several data centers and we want to have higher disaster recovery.  We actually want to use all of them.  For that we have multi-site, our application.  So you can take two-step clusters and you can configure them.  And there will be a sequence application on one of each other.  It can be active, active or passive active.  The metadata operation, which is use the equation, use the solution, bucket question, bucket delivery, and a few bucket metadata operations,  are sequenced because they are really important stuff and we cannot have a difference in those.  So we have a meta-master and in case he fails, you need to do failover and configure a new meta-master.  But the data is completely asynchronous. That means all different clusters, you cannot build the object and there will be replicated automattically to the allocasters.  A zone is what we call a self-cluster.  So if you have two locations, each one will have a self-cluster that will be two zones.  A zone group is just a group of zones that share the same data.  So even not in the same location, they will be in the same zone group because they will be replicated from each other.  We have all multi-site support in HANA. It's called a sync agent. It's only active passive.  But from June, everything is active active.  I don't have time.  Okay.  More stuff.  So we talk about object life cycle. We don't support the full S3 object life cycle.  We have object operation and basic storage policy, which means that after the object expires, we move it to a different called storage.  We support object. We add more efficient object called the operation. We don't actually copy the data, but just update the object head to point.  So now it points to the same data as a different object.  We have a question. Thanks to me.  So we object to our trip. We have compassion. Also, we want to work with support.  We have static website. Anyone wants to use that website.  We have support. That's new in Khafen.  It's not active. We have met a little search.  We can do exporter data to elastic search and that's where the user can search the metadata.  And we're working on doing online backlit resharing.  I hope I have time to questions.  No.  Okay. Questions.  One.  Yes.  Maybe what is the performance parameter for the rather state-based data?  Well, you ask what's the performance parameter of using rather state-based data and not rather state-based data.  So we need to understand that we are very different.  Rather, this is REST API.  So REST API is just a PPP. That's low. It's a slow protocol like PPP.  So I think that's the main performance parameter.  But if you're talking about really, really large objects, then I think the REST API is limited.  We have some of the REST.  We are not as large as Bench because we have per-object also.  We have a few new objects, not just an object.  So I don't know the numbers, but those are currently.  Thank you.
Introduction_to_Ceph_cloud_object_storag_0_47.png Introduction_to_Ceph_cloud_object_storag_48_131.png Introduction_to_Ceph_cloud_object_storag_132_695.png Introduction_to_Ceph_cloud_object_storag_696_1811.png Introduction_to_Ceph_cloud_object_storag_1812_1955.png Introduction_to_Ceph_cloud_object_storag_1956_3383.png Introduction_to_Ceph_cloud_object_storag_3384_7307.png Introduction_to_Ceph_cloud_object_storag_7308_11507.png Introduction_to_Ceph_cloud_object_storag_11508_12095.png Introduction_to_Ceph_cloud_object_storag_12096_13379.png Introduction_to_Ceph_cloud_object_storag_13380_15251.png Introduction_to_Ceph_cloud_object_storag_15252_18071.png Introduction_to_Ceph_cloud_object_storag_18072_18167.png Introduction_to_Ceph_cloud_object_storag_18168_18707.png Introduction_to_Ceph_cloud_object_storag_18708_19403.png Introduction_to_Ceph_cloud_object_storag_19404_21863.png Introduction_to_Ceph_cloud_object_storag_21864_23315.png Introduction_to_Ceph_cloud_object_storag_23316_26255.png Introduction_to_Ceph_cloud_object_storag_26256_27947.png Introduction_to_Ceph_cloud_object_storag_27948_29735.png Introduction_to_Ceph_cloud_object_storag_29736_31223.png Introduction_to_Ceph_cloud_object_storag_31224_34823.png Introduction_to_Ceph_cloud_object_storag_34824_34943.png Introduction_to_Ceph_cloud_object_storag_34944_35075.png Introduction_to_Ceph_cloud_object_storag_35076_35207.png Introduction_to_Ceph_cloud_object_storag_35208_35459.png Introduction_to_Ceph_cloud_object_storag_35460_35975.png Introduction_to_Ceph_cloud_object_storag_35976_36227.png Introduction_to_Ceph_cloud_object_storag_36228_36359.png Introduction_to_Ceph_cloud_object_storag_36360_36503.png Introduction_to_Ceph_cloud_object_storag_36504_36743.png Introduction_to_Ceph_cloud_object_storag_36744_37127.png Introduction_to_Ceph_cloud_object_storag_37128_37511.png Introduction_to_Ceph_cloud_object_storag_37512_38015.png Introduction_to_Ceph_cloud_object_storag_38016_38147.png Introduction_to_Ceph_cloud_object_storag_38148_38279.png Introduction_to_Ceph_cloud_object_storag_38280_38399.png Introduction_to_Ceph_cloud_object_storag_38400_39035.png Introduction_to_Ceph_cloud_object_storag_39036_39551.png Introduction_to_Ceph_cloud_object_storag_39552_41831.png Introduction_to_Ceph_cloud_object_storag_41832_42743.png Introduction_to_Ceph_cloud_object_storag_42744_45791.png Introduction_to_Ceph_cloud_object_storag_45792_46283.png Introduction_to_Ceph_cloud_object_storag_46284_48347.png Introduction_to_Ceph_cloud_object_storag_48348_50375.png Introduction_to_Ceph_cloud_object_storag_50376_50435.png Introduction_to_Ceph_cloud_object_storag_50436_51371.png Introduction_to_Ceph_cloud_object_storag_51372_52499.png Introduction_to_Ceph_cloud_object_storag_52500_54887.png Introduction_to_Ceph_cloud_object_storag_54888_54935.png Introduction_to_Ceph_cloud_object_storag_54936_55871.png Introduction_to_Ceph_cloud_object_storag_55872_55967.png Introduction_to_Ceph_cloud_object_storag_55968_56327.png Introduction_to_Ceph_cloud_object_storag_56328_59243.png Introduction_to_Ceph_cloud_object_storag_59244_61283.png Introduction_to_Ceph_cloud_object_storag_61284_61703.png Introduction_to_Ceph_cloud_object_storag_61704_61823.png Introduction_to_Ceph_cloud_object_storag_61824_62471.png Introduction_to_Ceph_cloud_object_storag_62472_62591.png Introduction_to_Ceph_cloud_object_storag_62592_62723.png Introduction_to_Ceph_cloud_object_storag_62724_62855.png Introduction_to_Ceph_cloud_object_storag_62856_63071.png Introduction_to_Ceph_cloud_object_storag_63072_63119.png Introduction_to_Ceph_cloud_object_storag_63120_63192.png
This is Orit from the CoreSafe team. We can talk a little bit about our object storage, the RATOS Gateway in SEP. Hello. Thank you for coming so early on Sunday. I know it's hard. I'm Orit. I'm part of the SEP team in RATOS. And I work on Cloud of the Storage in SEP. And today we're going to explain what Cloud of the Storage is, how it's different to regular storage types, a little bit about SEP. So first, who uses SEP? Not much. We need more hands. So we explain what SEP is, how it's built, and then we talk about for this gateway, who is the part of SEP that provides Cloud of the Storage interface. And of course we have time for a few questions. So in general, I prefer people to ask questions during the presentation, so if someone needs a question, just raise your hand. So let's start. What is this Cloud of the Storage? So we have a block stage. Anybody who uses block stage? Yeah, of course. We all do. So the stage is divided into those fixed blocks depending on the device. We then have any metadata related to the data. You just write the data somewhere in the device of SEP left. And the application needs to manage to know what exactly it written there, where it's written. You can have several devices. But it's really fast. Well, first of all, it. And today we're going to say it's really fast. We have the other protocol, fibercharm, SCSI, SATA, iSCSI, and some people even use fibercharm over internet. But using block storage really hard for applications. So then, can file systems. So here the data is organized. We have higher fee of directories that contain files and other directories. We have users. We have metadata for this data. The file can be in any place. You can override it in the middle and allocate on demand. It's much easier for the application to use the storage. But it's not in block. That's a argument. Sometimes five system people, but it's slower. Because of the all metadata and you are here on the link. You can server not all files systems, but many have shown semantics. So you actually can log the file for read only or completely for you. And two different applications can use and write to the same file at the same time. We have local file system protocol like XC4, XFS and so on. And we also have network file protocol like NFS and SMB. And F2 is the Apple protocol by the class they decided not to use it anymore. And they're moving to SMB. But then came something new. It was the cloud. So we had that problem before the cloud came. We saw the story is going and going. So people moved to more distributed storage systems. And writing a distributed file system is really hard. I don't know if many of you have tried writing a regular file system hard, but when you try to distribute it, the metadata, you need to think it all around, it's a problem and there are a problem. The cluster people are you can say. But in the cloud it became even harder because we're not only talking about a large amount of data that it accessed all around, but we're talking also a large, really big data chunks. And then came the object storage that existed before the cloud object storage. But it was mostly like a niche, I guess, and I did research stuff on object storage, but it wasn't user-doer-standard. But nobody used it. But then came Amazon, actually. And when the site was on, object storage collapsed. I remember it was something like, I guess, ten years ago when S3 was launched, and I thought, wow, this is the way to do storage. How can we do that? I thought it really was amazing as a storage person. I was doing my research and actually said, no, we need to do something like writing an IBM. But of course, IBM is not good at being innovated and they never did anything like that. So the idea is very simple. One, we're talking about cloud environment. So you want the API to be really for cloud. So that means it should be HTTP-based, so that's West API. And you learn some of the amazing of the data, but the complex IRHF-like systems have is really hard. So it's just two levels. Amazon formed them buckets, Swift called them containers, pools. So just the container of objects is the way to organize your object into groups. And maybe add some properties for that. So that's bucket. And inside the bucket, you have those objects. An object are not just data. You have a name, you can add them all sort of metadata. Some are fixed and some you can decide on your own. So you can let them find them and know what's the data in them. You have users and tenants, which is like an umbrella for user to share the same data between them. You have authentication and ownership, access control list. And you talk about large objects, really large. So that means you need lots of storage. Because there's a large object, usually objects are immutable. That means you write them once, if you decide to change them, you need to actually override all the objects. You cannot write in between. And it's much more efficient for those large objects to be immutable and it simplifies the allocation. So the protocol today, the most common one is Amazon S3. It's the richest one. And I don't know what scale Amazon is, but I'm sure it's huge. I cannot imagine how much storage S3 has. In AppStack we have Swift and there's also Google Cloud Storage. So this is the ground around the API links. Put writes an object, get... This is a bucket. So put creates an object, bucket, get with the bucket. You have actually... it can be more complex because you can read also the data that are using head. You can delete a bucket, create an object. But it's not a simple copy of version. This is a simplification because as you see there's some headers there. Usually there's lots of header that says all sorts of metadata or maybe make the operation more complex. So we talked about really large objects. Anyone here has a downloaded stuff from the internet and you lost connection and you need to download. Everybody. It's really annoying. And then we talked about larger books. We can get to jiggle size. This is can be really hard. So for that we have the multi part. You want to upload a download an object but you want to maybe stop it, maybe we try. So we take a big object, you divide it to small parts and each part is being uploaded and downloaded parallel so you can handle network problems. And then you can continue where you stopped. And you can stop start, upload download. And also it's good for streaming for example. When you don't know the size of the objects you're actually going to upload or download. Download usually knows but upload sometimes generate the data when you upload it and for that you use multi part. But it's harder for the storage because until you finish your upload you don't know what the object will be and we cannot commit it. So we need to start all those temporary parts. And sometimes you have failure and we start the upload completely again. So we need to in the background to clean all the extra stuff we got. Another feature which I really like. So here we have used here VMS. So remember VMS, we have the file versioning. When you write on the file you get a dot something. I really like it because it sometimes do by mistake the leaf stuff. It will be good, you always have the previous copy. So again in cloud storage we have versioning, where we fold buckets, it's per bucket. But if all buckets are not with versioning but you can enable it, that's mean. When you override the object it creates a new object with a different version. If you delete the latest you will still have the old version. It's really helpful when you delete stuff by mistake or overwrite by mistake an object. And cloud objects are really cool stuff. I didn't talk to you about authentication because every protocol does it completely different. The main difference between Google Cloud Storage and S3 is authentication and it's incompatible. So it's different. Some use user passwords, some use user and a key and a secret. Everything is really different. So authentication is complex and we have object lifecycle. For example you can set a date on the object and when it gets to that date, called object exploration, that's what can be deleted automatically or moved to cloud storage. And lots of cool features. Any question about cloud storage? So let's talk about S3. So S3 feels so close to the circle pod which is the family of bucket poses in school. So all our versions are called one animal that name. So the latest stable version is dual. So self is open source. This is our give up. We have set all under it. It's all the self code. It's not only self. We have our own testing framework called totality. It's also open. And we have additional tests like we have S3 tests. There's S3 compatibility. We have other components of self. So self is open source. We are in problem. So it's open source, LGPL. It's software defined storage. Well, we are in the software defined storage room. That's mean it can run on many kind of platforms. You can configure and adapt it to different apologies and scale. It's a bit system and like you say, it doesn't have any single point of failure. When you have a large number of nodes, you cannot handle a single point of failure. Because you assume everything will fail. Large scale means failure happens. It's massively scalable. It was built to be large scale. So we serve PLC and build three nodes, but the minimum is five. And I actually recommend more. It's a large scale system, not for two nodes, three nodes. Because we are large scale, we assume that we have to either replicate data or use the words according to be more efficient space. Because we assume they don't have much left, so we need more copies. It's a large system. Failures happen. So we want self-healing because you cannot handle those failures manually. Everything should happen automatically. And we have unified access. So we have self-healing. It's the file system access. It's a positive client file system. You can use our client fields or our kernel client. And there's integration to Opus.Memilar. And you can use NFS Ganesha with it for NFS and Samba for SMB. And it is the block interface. Many know it from Opus.Tap. It's integrated completely with QMU. So you can use it with QMU for Xen or QMU for KVM. And you can use the kernel client if you want your own look device. And we have BodeSketway, which is a component that provides cloud object storage. Even if this all were WebOS, and I'm going to speak bit about WebOS. WebOS is a reliable, autonomous, distributed object storage. So as you see, it's an object storage, which is interesting. So when self-started, it was to build a very fast, scalable, distributed file system. And the style would rather us to be as the rich object storage that you can build the file system on it. Easier, please. But I can distribute a file system. It's never simple, even if you have distributed objects storage underneath. And in the meantime, we took to develop CFFS, the style of IBD, and WebOS Gaitway. And now we also have CFFS as product. So what this does, it does all the distributed stuff. So it does all the replication and all the ways you're coding. It's a flight object name-stance, and we call, we have a pool of objects. And you can configure it to pool. It could be a pool of very fast storage, or pool of slower storage with high-risk. It could be different applications. It could be freeways, or it could be res-accords. CFFS is a strong ecosystem system. And it's software-defined, that means it's aware of the infrastructure and the topology, and it matches two different architectures and systems. For placement, we have a HustBase algorithm with no lookup called Crush. And we want performance, so we serve the data directory to the client. So we wrote about Crush. Crush is our placement algorithm, and we wanted an algorithm because it's a large-scale system that doesn't require any lookup. Lookup requires some central area. You keep the table of the lookup. You can replicate it, but it's not very efficient in large scale. It's a bottleneck. So Crush doesn't require any lookup. It is topology aware, so that means you can actually put in cloud. You can say, this is work, and I don't want to place two copies on the same work. So you choose nodes that are not on the same work. You can configure the application. You can do weights. And it's really fast-capsulation, deterministic, and it's evenly distributed. That's why we call it Soderander because that's where we can benefit from all nodes. So a surf wireless cluster has two kinds of nodes. The first one is the object storage device. Those we have lots of nodes, demons. Tens to ten harvests. I think ten harvests is a lot, maybe thousands. And we have one OSB per disk. You can put an OSB on a wave, but it's a waste because we do the application ourselves. There's no reason to do additional replication in the storage layer. And its main thing we do is set the object to the user. It's a smart storage node that's meaning that's peering. And in case of a failure, it does the recovery automatically. There are always these talking with each other and we'll work with KDN. If a new node comes, it will rebalance the storage. The archive of nodes we have is the monitor nodes. They store all the clustering logic. They maintain the cluster membership. So if a node fails, the monitors get a notification because it still notifies that the OSB is not responding. And it will notify the monitor there's a failure, and then the monitor will spread it to the other OSBs. And when you add an OSB in the same way, we have a small number of those nodes, three, five, so on. They do a list of boxes. It doesn't stop any data to the user. It's just for the cluster membership data. So we have this cool object storage cluster. We want to accelerate. So we do need webis. And you actually can use webis if you have your own application. And it really needs lots of performance and you're willing to interact with the webis API directly. And you can use webis. Actually, some users will use webis directly. It has been into C++, a web will be Python. I don't remember. Probably more. And it's a very rich API. It's not just a sweet, wide object. You have the key value store. We call it OMAP. So inside one object, you have a key value store. And you can use this to store. For example, the five system stores. The director is in an OMAP and we'll just get where the bucket index is OMAP. It's a single object operation so you can update attributes and keys and data in one operation. It has snapshots for objects. Objects are mutable. That means you can actually partially override objects. And webis clusters are very similar to storage procedures in a database. You can actually write code that will run directly on the OSD when an object is changed. It's really a strong feature. It allows lots of performance. We use it all around the services. And we have a watch notify service so we can register an object. And when the object changes, we get a notification across the cluster. Everyone registers notification. So it's a very strong API, but it's not a standard API. It's not block interface, file, object, cloud object. It's your one. And if you really import a performance and you really need to make the effort, you'll be really cool. Any questions about self in German? Just a few words about self in English. Okay. So first, all these are divided into placement groups. And as we all have, we need to replicate one another. One can be the buster in some cases. This other case is just a replica. So they talk with each other and they can detect when there's failure. Because they need to talk with each other and then if they get in error, they notify the monitor. And then the cluster starts to under that case. So you need that FD that failed. It's taken while to mark it as failed. But when it's failed, you know all the data it contains. And it will be replicated to other SDs. So then they do it automatically. One more question about each object. Each node has a complete set of the whole object. Well, depending on the size of the object. But our maximum size of object in the radios is 4M. So 4M will be, let's say you have freeways replicates. So freeways will have those 4M. If the object is larger, then it's more complete. My question is, is the object distributed? That it means you have to have a minimum set of storage nodes to have to retrieve the complete object. No. To synthesize? No. You need just one. Yeah, you need one to read. To write, you need three. If it's freeways, you need three copies. But if you read, you need one copy. Yes? Is it better for set many lower quality machines, or is it better to have fewer number, but higher quality machines? Many lower. If that's the demand, of course we prefer many high quality. But the more you have machines, the more you can do large scale. Now we're going to talk about more about FEDIS Gateway, which is the component in FEDIS that provides the cloud of the search. So all the components of FEDIS Gateway uses Li-Ve-DOS to communicate with the radios clusters. So we have these clusters, voids, ds, and monitors, and that's where the radar is stored. And this gateway is a service built on top of that. So wireless is object storage. So that's good. It should be simple. We can have cloud object storage. But life is never simple. So it's not just to provide the rest of the API and matching the API we have. When we look at our object for radios and other security way, or cloud, we see some difference. So first radios, the biggest size of an object is four mega, and we're talking about cloud, so we talk about three really large objects. So we need some way of taking bigger objects and dividing them to those four mega-size objects. So radios objects are mutable, but in cloud we talk about immutable objects. The hardest part is that the poles inside of radios are not indexed, but user want to list all the objects in the bucket sorted by name. That means we need to add some indexing for the bucket to allow those listing. And radios as per pole are CL. So pole is like a bucket, but we need a CL as per object. So even in spiral-structured radios, is object, we still need to add layer to implement the cloud of the storage at the top of it. So we have the radios cluster. We save all the data inside the radios cluster. Radios gateway is stateless, so in case you see that you have lots of actions and operations, and the radios gateway is loaded, you can just run another instance of radios gateway on the same set cluster and both can work together and continue. And that way you can scale up. So again this is object storage. We have users and tenants. We have buckets, objects. We have metadata, per bucket, per object. We have access control list, per bucket, per object. We have authentication, which is really complex because we support server protocols. Today we support two-row cloud of Rixx protocol, S3 and Swift. In many cases you can use both in the same time, but sadly they are not compatible, so there are cases that it won't work. If you can't upload multi-part with Swift and with it, we'll mess with it because they sign differently the calculator, the signature differently. Versioning is very different. You cannot use both. Version bucket for S3 would not be versioned when you access this from Swift and other way. The authentication of course is very different, but we do support Keystone for S3, so that's maybe okay. We also support NFS. And it says that NFS is a file protocol. Yes, it is. It's not a full NFS. If your main work order is a file system, the CFFS, it's for users that mostly use object storage and they can use the NFS. We use NFS Gimisha on top of a library called libhgw to allow that. And it's for migrating from NFS to object storage or exporting a bit. And also if you have one, this legacy application that made this NFS very basic stuff, then you can use it. But for full-scale NFS, you use FFS or other NFS stuff. So we try to build the latest gateway as very accessible. So a request goes for all the layer one after the other. We have two layer, three parts that everyone can access all the time. I'll go a bit about those. So first we have the frontend. Frontend is what provides the rest API actually. So it has to be HTTP. So with two layers, the first is the old and un-recommended way. So we support fast-synth VGI with Apache and probably other web servers. The reason it's not recommended is because fast-synth GI is lots of security issues. But sometimes if you already have your own Apache and you handle the security issue with fast-synth GI, it can be an option. Today we have also, from Hama, five-flat actually, I don't remember. We have Sivot Web. It's in a very, very server-inside of the web disk gateway. So we recommend that. Then we go to the rest layer. It's actually the layer that converts from past the dialect between the protocol. So it passes S3, S3, S3, then RAPI. Then it goes to execution layer, which is the common layer for protocols. We don't run to IDECOR twice. So we try to use as much as common code, handling different protocols. The next layer is the layer that talks to Redis. It's not enough to use Redis. We need to do some stuff on our own. So for example, we do object striping, atomic overrides, and all the bucket index handling. And we also, it contains the object classes, the run and the SD. We have quarter, and we have quarter per user, quarter per bucket. We have the execution layer because in many cases, in many layers, we need to attend, sometimes we need to educate again. So we support AWS-free, AWS-free, Keystone-2, Keystone-B-3, LDAP. This is what we're going on on this US, but hopefully it will be in the end of CoC or maybe in the winners. Objects are large, so we don't delete them and free the space immediately because it takes time. So it will move to a garbage collection in the background as a process that contains the large objects. I don't have time, so I'm going to skip some slides. So this is our rebuild object. We take an object with a head object and pull. All objects have a head object. It contains all the object metadata it attributes. Up to 512 cases of data. Small objects only have an head. Large objects are stripes and several tail objects. To have a fast access to objects, objects are names inside of this. So they have the number 123, the bucket ID. It's not the name because bucket can be named bucket, so it's a unique ID for bucket. And then the object name. That means that we don't need to do any lookup in order to read the head because it contains the metadata and we want many cases to be able to do it fast. And then the tail is the bucket ID and then some UUID and the pot. A little bit of bucket index. So bucket index is just all the objects in the bucket sorted by name. But when we talk about clouded storage with cases that one bucket contains millions of objects. And then the bucket index is also a bottleneck for performance and also because it's in the end stored in an older weather subject. If we pass one object line, it's also inefficient. So first thing, if you have lots of larger objects, the search number is up to whatever K object. One bucket in this, that's fine. If you have more than one under case, then you need to use shouting. We collect the bucket index and split it to several objects. Depending on the number of objects, you decide the number of shards you want. We have now support for offline reshouting. So in case you didn't reshout the bucket and suddenly you got to live number of objects, you can reshout it. I think it's in the summer, the summer, the 10, and it's in the new versions. And we're trying to work on the online reshouting that you will need to offline the bucket and stop pirating it. You can do it with Ion. We skip that. So our quota is the redistributed system. Objects can be written from many, many places, several radios get ways. We need some way to coordinate the quota. So we use the radiosubject class objects to do that. So when we actually write the objects in the RSD, we update the quota. That means the quota is not completely consistent with sometimes a myth a bit. But I think five systems have the same issue. We have the metadata cache and we use watch notify to handle that. That's mainly in case the metadata change, the other way is get an application and then can invalidate that entry from the cache. So multi-site. So we're talking about cloud, but cloud is never one data center. Many times we want several data centers and we want to have higher disaster recovery. We actually want to use all of them. For that we have multi-site, our application. So you can take two-step clusters and you can configure them. And there will be a sequence application on one of each other. It can be active, active or passive active. The metadata operation, which is use the equation, use the solution, bucket question, bucket delivery, and a few bucket metadata operations, are sequenced because they are really important stuff and we cannot have a difference in those. So we have a meta-master and in case he fails, you need to do failover and configure a new meta-master. But the data is completely asynchronous. That means all different clusters, you cannot build the object and there will be replicated automattically to the allocasters. A zone is what we call a self-cluster. So if you have two locations, each one will have a self-cluster that will be two zones. A zone group is just a group of zones that share the same data. So even not in the same location, they will be in the same zone group because they will be replicated from each other. We have all multi-site support in HANA. It's called a sync agent. It's only active passive. But from June, everything is active active. I don't have time. Okay. More stuff. So we talk about object life cycle. We don't support the full S3 object life cycle. We have object operation and basic storage policy, which means that after the object expires, we move it to a different called storage. We support object. We add more efficient object called the operation. We don't actually copy the data, but just update the object head to point. So now it points to the same data as a different object. We have a question. Thanks to me. So we object to our trip. We have compassion. Also, we want to work with support. We have static website. Anyone wants to use that website. We have support. That's new in Khafen. It's not active. We have met a little search. We can do exporter data to elastic search and that's where the user can search the metadata. And we're working on doing online backlit resharing. I hope I have time to questions. No. Okay. Questions. One. Yes. Maybe what is the performance parameter for the rather state-based data? Well, you ask what's the performance parameter of using rather state-based data and not rather state-based data. So we need to understand that we are very different. Rather, this is REST API. So REST API is just a PPP. That's low. It's a slow protocol like PPP. So I think that's the main performance parameter. But if you're talking about really, really large objects, then I think the REST API is limited. We have some of the REST. We are not as large as Bench because we have per-object also. We have a few new objects, not just an object. So I don't know the numbers, but those are currently. Thank you.
is, how it's built, and then we talk about for software defined storage. Well, we are in the which is a component that provides cloud object reliable, autonomous, distributed object storage. support two-row cloud of Rixx protocol, S3 and different. Rather, this is REST API. So REST API