HTTP tunneling in Go using HTTP/2 streams
Galeiras, Iago López
This talk describes our experience developing Wormhole Connector, a distributed proxy component that connects external enterprise systems to a Kyma Kubernetes cluster. The connection between the Wormhole Connector and the Kubernetes cluster is based on HTTP/2, taking advantage of the stream concept to multiplex multiple connections (HTTP/1 or HTTP/2) through one active HTTP/2 connection. It will cover how we use the Serf and Raft libraries to make the Wormhole Connector highly available, our experience using the Go standard library to support HTTP/2 connections, and technical details describing how everything works under the hood. This talk describes our experience developing Wormhole Connector, a distributed proxy component that connects external enterprise systems to a Kyma Kubernetes cluster. The connection between the Wormhole Connector and the Kubernetes cluster is based on HTTP/2, taking advantage of the stream concept to multiplex multiple connections (HTTP/1 or HTTP/2) through one active HTTP/2 connection. It will cover how we use the Serf and Raft libraries to make the Wormhole Connector highly available, our experience using the Go standard library to support HTTP/2 connections, and technical details describing how everything works under the hood.
 Yeah. So I'm Jago. Hi. I'm a tech lead and co-founder at Kimfolk. I guess you've heard about Kimfolk, being that you are at this conference and there's a logo everywhere. But yeah. Basically we are a software consultancy. We're a team of software developers that help companies to work on their open source projects when they don't have resources or they don't have the expertise needed. But recently we have a bit of a shift. So we forked CoreOS container Linux and we called it Flatcar. And we are offering support for that. And soon we'll be announcing some other product related to Kubernetes. So stay tuned. Anyway. The plan for this talk is as follows. I will first introduce a problem that we were trying to solve. Then I will give a high level description of the program that we developed to solve this problem. I will talk a bit about the technologies used, surf raft and HTTP2. And then I will share some go re-implementation details that I found interesting. I will try to do a demo. Hopefully it will work. And then I will talk about what's missing and give some conclusion. So the problem is basically there's this enterprise systems that have a lot of legacy code and they're running in big monoliths or something like that. And they want to communicate with a Kima cluster. So Kima is this product by SAP. So it basically gives developers ways to easily deploy their applications in a Kubernetes cluster. And it gives some kind of serverless framework by default so they can use lambda functions and stuff like that. But for the purpose of this talk, just think of a regular Kubernetes cluster because I actually haven't even run the Kima cluster. I'm just dealing with playing Kubernetes. So they want to connect that cluster with some external services and they want to do it in a transparent way. So they want to be transparent to client applications and they want to reduce network latency between the cluster and the external system. And they also want to provide high availability and resilience in a simple way. So you can just deploy your application to this external system and you don't need any Kubernetes or any fancy distributed system. They just need something simple that can provide high availability. So yeah, I will give now a high level description of the wormhole connector, which is what we call this project. So basically it's just a distributed proxy. So being distributed, it provides high availability and if a node goes down, it can continue fine. It's based on using surf and raft, which are two distributed algorithms, which I will talk about a bit more later. And yeah, it connects the enterprise systems to Kima and Kima to enterprise systems. And it uses, to do that, it uses an HTTP tunnel between the enterprise system and Kima. So there's two components of this. One is the wormhole connector proper that runs on the enterprise solution side. And it's just a basic HTTP proxy. So services running on enterprise solution can talk to the Kima cluster by just setting the proxy variables there. So all the connections received from the enterprise solution services that will go through this tunnel to the Kubernetes cluster. And you can find a project there. The other component is the one that runs on the actual Kubernetes cluster. And what it does is it proxies the received connection from the tunnel to the services inside the cluster. But it also does the other way around. So it's also an HTTP proxy. So services inside the cluster can also talk to the enterprise system. And yeah, this is, yeah, it has some small example implementation there inside the same repository. So I have to say that this project was more like a proof of concept. So don't expect like a finalized software. So yeah, just a heads up. So I want to talk a bit about the technologies involved, surf and raft first. So yeah, surf is a Go-Sip distributed protocol. And yeah, it basically allows you to have some cluster membership and to pass some messages to all the members of your cluster. So you can see there in this GIF I found on the internet that some node receives some kind of message and it goes to the other nodes because one node talks to another node and then the other node talks to others. So in the end, the whole cluster will know this message that is being sent. Yeah. So we use this in the wormhole connector to basically add new members. So basically you just need to start the wormhole connector and tell it, okay, here's another member and then it will join the cluster. And yeah, it also provides a way to know when a member has disconnected to remove it from the cluster. And then we use raft, which is a consensus algorithm. It's used by things like etsyd console. And basically with raft, you have some certain state that you want to have in your cluster and you want to have it consistent. So if you query other members of the cluster, they have the same view of the state. And yeah, it provides a leader election. So you can have a leader that is doing all the right operations and then the others are followers who can, you know, you can read from them and get the same state. So in wormhole connector, we use that to elect a leader, of course. And we start some state in the raft SSM, which is finite state machine. And yeah, for now, it's just a simple array, but this has some application in this context because there's this concept of event bus events. And yeah, basically this legacy system can send events to the cluster. And we have to save the state there somewhere in the cluster. So we can have retries and we can have, yeah, things like that. In the future, you will start those events I was talking about, but for now it's just a simple array of strings because as I said, it's a proof of concept. Yeah, so if you want to learn more about drafts in a more detailed way, there's a lot of good resources on the internet. There's this page that has some nice visualizations and explanations. And of course, you can also read the raft paper, which is pretty good. Yeah, so as I said, I just implemented a simple queue. And you just send an event to one of the members here, connector one, for example. And then you can read it from another member and you will see that the same event is there. You can also delete the events. And yeah, that's basically the toy API that we've implemented. So talking about some other technology, HTTP 2. Basically, this is just the main differences between HTTP 1 and HTTP 2. So HTTP 2 provides header compression, so things are more efficient. And it's a binary format, so it's also a way to improve efficiency. And the most important part for us was that it has request response multiplexing. That means that with a single TCP connection, you can have several connections, HTTP connections, and they don't need several TCP connections like with HTTP 1. Yeah, so it's something like this. You have some HTTP 2 connection, which is a TCP connection between the client and the server. And then you have different streams. So this can be totally different data. And it will reuse the TCP connection, so you'll have to do the handshake again or the HTTP as negotiation. It's just a tunnel that stays there. So I will talk now about some implementation details of this. Yeah, first of all, the libraries I've used are the HashiCorp ones. And they were pretty cool because they were simple to use. Basically, for example, for SERV, you just have some default config. You configure some values like the bind address you're listening on and stuff like that. And you just create a SERV instance. And then you just get the events when a member joins a cluster or leaves a cluster. So that was pretty cool. And for RAV, same thing. So you need some kind of store on disk. We've chosen the bold DB store because of simplicity. And you just create a new store, create a snapshot store which can be backed by a file, for example. You need to choose a transport which shows the simple TCP. And then you need to give this finite state machine implementation. I'll talk about that in the next slide. And then you just start the RAV object. And yeah, that's pretty much it. So it's pretty easy to use. And yeah, well-documented. So this FSM thing, oops, sorry. This FSM interface is just some abstraction that the RAV library has. So you can have a finite state machine of your own data structure. So in this example, I just have this Go structure which has some slides of events which are just simple strings. And then some mutex to avoid problems with concurrency. And then you have to implement these operations. So yeah, you can change this structure and store whatever you want. And this state will be replicated through all the members of your cluster. So that was pretty neat, too. One thing to mention here is that the write operations are only, you can only do it on the leader. And that simplifies things for the protocol implementation. So what we do basically is just, if there's a write request, we redirect to the leader and then the request will be written. So this is the basic architecture of, like the full picture of this thing I was talking about, this project. So you see here that there's a wormhole connector and it has several boxes. So it's this distributed part. So the part on the left is this enterprise system. And the part on the right is the Kima cluster or Kubernetes. So basically you see here two connections, one in red on top. That's between the client and some Kubernetes service inside the cluster. And basically the client just sets a proxy to connect through the wormhole connector. The wormhole connector will have a tunnel to the wormhole dispatcher running in the cluster. And then the wormhole dispatcher will do the dispatching of the traffic to the services inside the Kubernetes cluster. You also have the other way around. So you have a Kubernetes spot there in blue, I guess, that is also configured with a proxy to the wormhole dispatcher. And then the connection will be routed through the tunnel again back to the connector and connected to a server in this enterprise system. So yeah, that's pretty much the architecture diagram. Yeah, so some other details to be able to do this of only using one TCP connection. Golang has a very simple way of doing this. So you just create your HTTP client. This HTTP client contains some transport. And as long as you use the same HTTP client, your requests will be routed through this one connection. So this is a pretty neat API and it's easy to use. And yeah, it just works. We didn't have to care much about how. And then tunnels, yay. So this is some chunk of code on the slide. So I guess it's not so nice. But I can try to go through it. So basically we have here two things. One is if the connection going through the proxy is HTTPS, it uses this connect method for HTTP and creates a tunnel to the other endpoint. And this is so because we don't want to get the unencrypted information in the proxy. So we need to do a tunnel. On the other case, it's just a simple HTTP without encryption. So the proxy will just remove some headers and will just pass the connection to the other side and then we'll read the tunnels. And then, yeah, that's pretty much it. For HTTPS, we create this tunnel. And this tunnel will use the same client. So it will reuse the connection for the tunnel. And then, yeah, we have some details there. If it's HTTP one, you have to hijack the connection. If it's HTTP two, you cannot do that because if you hijack the whole connection, the other streams on the connection will die. So you do something else. But yeah, you get the idea. It was pretty easy. One other interesting thing we're doing is if I go back to the diagram, yeah, there's a connection from the dispatcher to the connector. But we don't want the dispatcher to know the address of the connector. So what we did here is we did some kind of reverse tunnel. So the connector will connect to the dispatcher and then the roles will be inverted and then the client will be the dispatcher and the server will be the connector. So this is on this way in Go. So basically, the dispatcher will accept some TCP connection. You can create a transport with that. And then you can tell the transfer to return the connection that was established before to be the transport. And then you can create a client and then you can just do get or whatever. On the other side, you basically, this is the other side. So the connector will dial to the dispatcher. And once you have this connection, you can just set up a new server and tell it to serve the connection on this TCP connection. So basically, you do the reverse tunnel. Right. So now I'm going to try to do some demos. So let's see if they work. Let me change my screen configuration. Hopefully I don't break any video. Maybe I did. Can you see anything, Daniel? Okay. Okay. Well, there's some. Yeah, there we go. All right. So, well, let's start with this other demo. So I'm going to start three instances of the wormhole connector and they're going to be running in rocket containers so I can have, yeah, just because I developed a lot of on rocket and it's just easier for me. Yeah, I guess this is a very long command, but this, and it doesn't really render properly. But this basically sets up a bunch of options, starts the wormhole connector process. And yeah, that's pretty much it. And now you can see that here it says the node is a follower and then it gets promoted to a leader because there's only one member in the cluster. So I will start some, a couple more telling it the address of one of the members. So yeah, now we have here two followers in the bottom and one leader here. So now I was just, I'm just going to show this API that I mentioned of a simple distributed queue. So I can just, I don't know, post some event, event one to the first, the leader. And yeah, that works fine. I can just get it from the same leader. I get here event one. Or I can get it from another node. It's the same result. And yeah, I can just post again another one, like event two. I can post it to, I don't know, another node, like three. And but for this I need to enable redirection, so dash L. And now if I get an event from any, I posted the same, I don't know. Yeah. So it still has the old event because I haven't deleted it yet. So let's do that. And now if we query again, we see event two. We can delete that too. And yeah, that's the same. So that's one demo of this FSM thing. Now I want to show another demo of the whole workflow. So I have a mini queue Kubernetes cluster running. And yeah, you can see that it's running this wormhole dispatcher service I was mentioning. So now I can start the wormhole connector. Just, yeah, I set up some entries in my Etsy hosts. So it's certificate work. But basically I'm connecting to the dispatcher. And now you see this connected to wormhole dispatcher. So this means that the reverse connection was established. And now what I'm going to do is, yeah, I have here an echo server service inside the Kubernetes cluster. And this just returns the some information about a request that you made. So it's just a simple ping pong kind of thing. But this is running on the Kubernetes cluster. So in theory I shouldn't be able to access it on my host. But if I set the proxy to be the connector wormhole.io, this is an address pointing to this wormhole connector and some certificates so I can trust the CA. Then I can just use this URL, which is the name of a Kubernetes service with this port. And I'll just get a response. You can see here that there's an outgoing HTTP connection. And this goes to the cluster and it connects to this service inside. Yeah. So I can do this several times. And then if I do net stat, you still see only two connections going to that Kubernetes cluster. One is the reverse tunnel and the other one is the other tunnel. So the other thing I can do is just run some pod. Okay. So I'm running an alpine pod. I'm passing the CA here in an m variable. So the thing is trusted. So hopefully this will start. It doesn't want to start. Still creating. I guess I'm downloading this thing on the network. Oh, yeah. And this is a nice bug about QCTL of my version at least. That if you have already an image of the same name running, I mean, a deployment of the same name running, it just crashes. That's neat. Okay. This for some reason doesn't work. But what this demo will show is that you can get to a pod inside the Kubernetes cluster, configure the proxy to be the dispatcher, and then it will connect to my host. I was going to show it by just running net cat here. But this doesn't seem to work. Oh, it's running. So hopefully I can attach to it. Good. So, yeah. Let me pull some instructions here because I don't want to... Yeah. Okay. Internet is not really working well. Yeah. Anyway. Trust me that this works. That's it. Okay. So the missing pieces here, yeah. One thing that you notice is that there was two TCP connections, one on one side and one on the other side. I would think that theoretically it's possible to use just one TCP connection and that was the goal in the beginning. But I didn't manage to do it in the time that we had to work on this project. Yeah. I'm not sure if with the Go API you can do this or not or you have to do some other weird stuff. So if you have any ideas about this, please let me know. Yeah. And the other thing that's missing is this events support that I was mentioning. So basically the enterprise solution will send some event to the cluster and we will be able to store the events and try to deliver them to the cluster. And if that fails, we can do retries and all that stuff. So that's not implemented. And even though this is a simple program that you can run to create a cluster on the enterprise side, we also want to support deploying that in Kubernetes. And for that, of course, you don't need any server for RAF because Kubernetes already provides the distributed primitives that you need. There's some working progress implementation, but yeah, we didn't have much time. So yeah, in conclusion, yeah. So we've implemented two clients of distributed system libraries, two proxies, two HTTP tunnels, a lot of two. We learned a lot of these things because this is not something we've worked on before. And we had a lot of fun. So yeah, I hope, yeah, you enjoy this. Thanks. And yeah. Do we have any questions? Yeah. So we have about five minutes for questions. I assume the proxy pod has to run in host networking. Sorry. No, they don't. They just connect to this Kubernetes service and they will get to that dispatcher thing. And yeah, that will get routed. And then connections into the cluster. So from the external cluster into the Kubernetes cluster. Okay. So which cluster you were meeting me for? We were talking outside, contacting the target Kubernetes cluster. Okay. That's running on the host. The server is running on the host network and then talking to the, okay. Yeah. I thought you meant the services in the Kubernetes cluster. It's kind of confusing. So there are no other questions. I have one. So how do you authenticate? Like how do you, the Kubernetes cluster and the enterprise cluster authenticate is just via HTTPS certificates? Although you have something additional on top of that? No, just HTTPS certificates. So basically, the authentication between the proxy and the client application is via HTTPS certificates, of course. And then you have to specify some CA that you trust for the Kubernetes cluster. And so the certificates for the Kubernetes cluster endpoint has to be signed by that CA. So like in regular Kubernetes. Yeah. Thank you for the talk. Thank you for the talk. So you have this thing where you do the reverse proxy and you basically don't accept on a listening socket. And how do you make sure that there's a connection ready when you need it? Sorry. How do you make sure that there's a connection ready to be accepted when you need to dial out essentially? Yeah. So it's basically just a loop that listens on a port. And if it gets a connection there, then yeah, it knows with some headers, it knows that it's from the wormhole connector. And then it does this logic of doing the reverse thing. I'm not sure if that makes sense. So it's just a very simple implementation. We didn't do anything fancy. Cool. So thanks. And yeah, enjoy the conference. Thanks, Jago. Thank you.
 Yeah. So I'm Jago. Hi. I'm a tech lead and co-founder at Kimfolk. I guess you've heard  about Kimfolk, being that you are at this conference and there's a logo everywhere. But yeah.  Basically we are a software consultancy. We're a team of software developers that help companies  to work on their open source projects when they don't have resources or they don't have  the expertise needed. But recently we have a bit of a shift. So we forked CoreOS container  Linux and we called it Flatcar. And we are offering support for that. And soon we'll be  announcing some other product related to Kubernetes. So stay tuned. Anyway. The plan for this  talk is as follows. I will first introduce a problem that we were trying to solve. Then  I will give a high level description of the program that we developed to solve this problem.  I will talk a bit about the technologies used, surf raft and HTTP2. And then I will share  some go re-implementation details that I found interesting. I will try to do a demo. Hopefully  it will work. And then I will talk about what's missing and give some conclusion. So the problem  is basically there's this enterprise systems that have a lot of legacy code and they're  running in big monoliths or something like that. And they want to communicate with a  Kima cluster. So Kima is this product by SAP. So it basically gives developers ways to easily  deploy their applications in a Kubernetes cluster. And it gives some kind of serverless  framework by default so they can use lambda functions and stuff like that. But for the  purpose of this talk, just think of a regular Kubernetes cluster because I actually haven't  even run the Kima cluster. I'm just dealing with playing Kubernetes. So they want to  connect that cluster with some external services and they want to do it in a transparent way.  So they want to be transparent to client applications and they want to reduce network latency between  the cluster and the external system. And they also want to provide high availability and  resilience in a simple way. So you can just deploy your application to this external system  and you don't need any Kubernetes or any fancy distributed system. They just need something  simple that can provide high availability. So yeah, I will give now a high level description  of the wormhole connector, which is what we call this project. So basically it's just  a distributed proxy. So being distributed, it provides high availability and if a node  goes down, it can continue fine. It's based on using surf and raft, which are two distributed  algorithms, which I will talk about a bit more later. And yeah, it connects the enterprise  systems to Kima and Kima to enterprise systems. And it uses, to do that, it uses an HTTP  tunnel between the enterprise system and Kima. So there's two components of this. One is  the wormhole connector proper that runs on the enterprise solution side. And it's just  a basic HTTP proxy. So services running on enterprise solution can talk to the Kima cluster  by just setting the proxy variables there. So all the connections received from the enterprise  solution services that will go through this tunnel to the Kubernetes cluster. And you  can find a project there. The other component is the one that runs on the actual Kubernetes  cluster. And what it does is it proxies the received connection from the tunnel to the  services inside the cluster. But it also does the other way around. So it's also an HTTP  proxy. So services inside the cluster can also talk to the enterprise system. And yeah,  this is, yeah, it has some small example implementation there inside the same repository. So I have  to say that this project was more like a proof of concept. So don't expect like a finalized  software. So yeah, just a heads up. So I want to talk a bit about the technologies involved,  surf and raft first. So yeah, surf is a Go-Sip distributed protocol. And yeah, it basically  allows you to have some cluster membership and to pass some messages to all the members  of your cluster. So you can see there in this GIF I found on the internet that some node  receives some kind of message and it goes to the other nodes because one node talks to  another node and then the other node talks to others. So in the end, the whole cluster  will know this message that is being sent. Yeah. So we use this in the wormhole connector  to basically add new members. So basically you just need to start the wormhole connector  and tell it, okay, here's another member and then it will join the cluster. And yeah,  it also provides a way to know when a member has disconnected to remove it from the cluster.  And then we use raft, which is a consensus algorithm. It's used by things like etsyd  console. And basically with raft, you have some certain state that you want to have in  your cluster and you want to have it consistent. So if you query other members of the cluster,  they have the same view of the state. And yeah, it provides a leader election. So you  can have a leader that is doing all the right operations and then the others are followers  who can, you know, you can read from them and get the same state. So in wormhole connector,  we use that to elect a leader, of course. And we start some state in the raft SSM, which  is finite state machine. And yeah, for now, it's just a simple array, but this has some  application in this context because there's this concept of event bus events. And yeah,  basically this legacy system can send events to the cluster. And we have to save the state  there somewhere in the cluster. So we can have retries and we can have, yeah, things  like that. In the future, you will start those events I was talking about, but for now it's  just a simple array of strings because as I said, it's a proof of concept. Yeah, so if  you want to learn more about drafts in a more detailed way, there's a lot of good resources  on the internet. There's this page that has some nice visualizations and explanations.  And of course, you can also read the raft paper, which is pretty good. Yeah, so as I  said, I just implemented a simple queue. And you just send an event to one of the members  here, connector one, for example. And then you can read it from another member and you  will see that the same event is there. You can also delete the events. And yeah, that's  basically the toy API that we've implemented. So talking about some other technology, HTTP  2. Basically, this is just the main differences between HTTP 1 and HTTP 2. So HTTP 2 provides  header compression, so things are more efficient. And it's a binary format, so it's also a way  to improve efficiency. And the most important part for us was that it has request response  multiplexing. That means that with a single TCP connection, you can have several connections,  HTTP connections, and they don't need several TCP connections like with HTTP 1. Yeah, so  it's something like this. You have some HTTP 2 connection, which is a TCP connection between  the client and the server. And then you have different streams. So this can be totally  different data. And it will reuse the TCP connection, so you'll have to do the handshake  again or the HTTP as negotiation. It's just a tunnel that stays there. So I will talk  now about some implementation details of this. Yeah, first of all, the libraries I've used  are the HashiCorp ones. And they were pretty cool because they were simple to use. Basically,  for example, for SERV, you just have some default config. You configure some values  like the bind address you're listening on and stuff like that. And you just create a  SERV instance. And then you just get the events when a member joins a cluster or leaves a  cluster. So that was pretty cool. And for RAV, same thing. So you need some kind of  store on disk. We've chosen the bold DB store because of simplicity. And you just create  a new store, create a snapshot store which can be backed by a file, for example. You  need to choose a transport which shows the simple TCP. And then you need to give this  finite state machine implementation. I'll talk about that in the next slide. And then  you just start the RAV object. And yeah, that's pretty much it. So it's pretty easy to use.  And yeah, well-documented. So this FSM thing, oops, sorry. This FSM interface is just some  abstraction that the RAV library has. So you can have a finite state machine of your  own data structure. So in this example, I just have this Go structure which has some  slides of events which are just simple strings. And then some mutex to avoid problems with  concurrency. And then you have to implement these operations. So yeah, you can change  this structure and store whatever you want. And this state will be replicated through  all the members of your cluster. So that was pretty neat, too. One thing to mention here  is that the write operations are only, you can only do it on the leader. And that simplifies  things for the protocol implementation. So what we do basically is just, if there's a  write request, we redirect to the leader and then the request will be written. So this  is the basic architecture of, like the full picture of this thing I was talking about,  this project. So you see here that there's a wormhole connector and it has several boxes.  So it's this distributed part. So the part on the left is this enterprise system. And  the part on the right is the Kima cluster or Kubernetes. So basically you see here two  connections, one in red on top. That's between the client and some Kubernetes service inside  the cluster. And basically the client just sets a proxy to connect through the wormhole  connector. The wormhole connector will have a tunnel to the wormhole dispatcher running  in the cluster. And then the wormhole dispatcher will do the dispatching of the traffic to  the services inside the Kubernetes cluster. You also have the other way around. So you  have a Kubernetes spot there in blue, I guess, that is also configured with a proxy to the  wormhole dispatcher. And then the connection will be routed through the tunnel again back  to the connector and connected to a server in this enterprise system. So yeah, that's  pretty much the architecture diagram. Yeah, so some other details to be able to do this  of only using one TCP connection. Golang has a very simple way of doing this. So you just  create your HTTP client. This HTTP client contains some transport. And as long as you  use the same HTTP client, your requests will be routed through this one connection. So  this is a pretty neat API and it's easy to use. And yeah, it just works. We didn't have  to care much about how. And then tunnels, yay. So this is some chunk of code on the  slide. So I guess it's not so nice. But I can try to go through it. So basically we have  here two things. One is if the connection going through the proxy is HTTPS, it uses this  connect method for HTTP and creates a tunnel to the other endpoint. And this is so because  we don't want to get the unencrypted information in the proxy. So we need to do a tunnel. On  the other case, it's just a simple HTTP without encryption. So the proxy will just remove  some headers and will just pass the connection to the other side and then we'll read the  tunnels. And then, yeah, that's pretty much it. For HTTPS, we create this tunnel. And  this tunnel will use the same client. So it will reuse the connection for the tunnel.  And then, yeah, we have some details there. If it's HTTP one, you have to hijack the connection.  If it's HTTP two, you cannot do that because if you hijack the whole connection, the other  streams on the connection will die. So you do something else. But yeah, you get the idea.  It was pretty easy. One other interesting thing we're doing is if I go back to the diagram,  yeah, there's a connection from the dispatcher to the connector. But we don't want the dispatcher  to know the address of the connector. So what we did here is we did some kind of reverse  tunnel. So the connector will connect to the dispatcher and then the roles will be inverted  and then the client will be the dispatcher and the server will be the connector. So this  is on this way in Go. So basically, the dispatcher will accept some TCP connection. You can create  a transport with that. And then you can tell the transfer to return the connection that  was established before to be the transport. And then you can create a client and then  you can just do get or whatever. On the other side, you basically, this is the other side.  So the connector will dial to the dispatcher. And once you have this connection, you can  just set up a new server and tell it to serve the connection on this TCP connection. So basically,  you do the reverse tunnel. Right. So now I'm going to try to do some demos. So let's see  if they work. Let me change my screen configuration. Hopefully I don't break any video. Maybe I did.  Can you see anything, Daniel? Okay. Okay. Well, there's some. Yeah, there we go. All  right. So, well, let's start with this other demo. So I'm going to start three instances  of the wormhole connector and they're going to be running in rocket containers so I can  have, yeah, just because I developed a lot of on rocket and it's just easier for me. Yeah,  I guess this is a very long command, but this, and it doesn't really render properly. But  this basically sets up a bunch of options, starts the wormhole connector process. And  yeah, that's pretty much it. And now you can see that here it says the node is a follower  and then it gets promoted to a leader because there's only one member in the cluster. So  I will start some, a couple more telling it the address of one of the members. So yeah,  now we have here two followers in the bottom and one leader here. So now I was just, I'm  just going to show this API that I mentioned of a simple distributed queue. So I can just,  I don't know, post some event, event one to the first, the leader. And yeah, that works  fine. I can just get it from the same leader. I get here event one. Or I can get it from  another node. It's the same result. And yeah, I can just post again another one, like event  two. I can post it to, I don't know, another node, like three. And but for this I need  to enable redirection, so dash L. And now if I get an event from any, I posted the same,  I don't know. Yeah. So it still has the old event because I haven't deleted it yet. So  let's do that. And now if we query again, we see event two. We can delete that too. And  yeah, that's the same. So that's one demo of this FSM thing. Now I want to show another  demo of the whole workflow. So I have a mini queue Kubernetes cluster running. And yeah,  you can see that it's running this wormhole dispatcher service I was mentioning. So now  I can start the wormhole connector. Just, yeah, I set up some entries in my Etsy hosts.  So it's certificate work. But basically I'm connecting to the dispatcher. And now you see  this connected to wormhole dispatcher. So this means that the reverse connection was  established. And now what I'm going to do is, yeah, I have here an echo server service  inside the Kubernetes cluster. And this just returns the some information about a request  that you made. So it's just a simple ping pong kind of thing. But this is running on the  Kubernetes cluster. So in theory I shouldn't be able to access it on my host. But if I  set the proxy to be the connector wormhole.io, this is an address pointing to this wormhole  connector and some certificates so I can trust the CA. Then I can just use this URL,  which is the name of a Kubernetes service with this port. And I'll just get a response.  You can see here that there's an outgoing HTTP connection. And this goes to the cluster  and it connects to this service inside. Yeah. So I can do this several times. And then if  I do net stat, you still see only two connections going to that Kubernetes cluster. One is the  reverse tunnel and the other one is the other tunnel. So the other thing I can do is just  run some pod. Okay. So I'm running an alpine pod. I'm passing the CA here in an m variable.  So the thing is trusted. So hopefully this will start. It doesn't want to start. Still  creating. I guess I'm downloading this thing on the network. Oh, yeah. And this is a nice  bug about QCTL of my version at least. That if you have already an image of the same name  running, I mean, a deployment of the same name running, it just crashes. That's neat.  Okay. This for some reason doesn't work. But what this demo will show is that you can get  to a pod inside the Kubernetes cluster, configure the proxy to be the dispatcher, and then it  will connect to my host. I was going to show it by just running net cat here. But this  doesn't seem to work. Oh, it's running. So hopefully I can attach to it. Good. So, yeah.  Let me pull some instructions here because I don't want to... Yeah. Okay. Internet is  not really working well. Yeah. Anyway. Trust me that this works. That's it. Okay. So the  missing pieces here, yeah. One thing that you notice is that there was two TCP connections,  one on one side and one on the other side. I would think that theoretically it's possible  to use just one TCP connection and that was the goal in the beginning. But I didn't manage  to do it in the time that we had to work on this project. Yeah. I'm not sure if with the  Go API you can do this or not or you have to do some other weird stuff. So if you have  any ideas about this, please let me know. Yeah. And the other thing that's missing is  this events support that I was mentioning. So basically the enterprise solution will  send some event to the cluster and we will be able to store the events and try to deliver  them to the cluster. And if that fails, we can do retries and all that stuff. So that's  not implemented. And even though this is a simple program that you can run to create  a cluster on the enterprise side, we also want to support deploying that in Kubernetes.  And for that, of course, you don't need any server for RAF because Kubernetes already  provides the distributed primitives that you need. There's some working progress implementation,  but yeah, we didn't have much time. So yeah, in conclusion, yeah. So we've implemented  two clients of distributed system libraries, two proxies, two HTTP tunnels, a lot of two.  We learned a lot of these things because this is not something we've worked on before.  And we had a lot of fun. So yeah, I hope, yeah, you enjoy this. Thanks. And yeah.  Do we have any questions? Yeah. So we have about five minutes for questions.  I assume the proxy pod has to run in host networking. Sorry. No, they don't. They just  connect to this Kubernetes service and they will get to that dispatcher thing. And yeah,  that will get routed. And then connections into the cluster. So from the external cluster  into the Kubernetes cluster. Okay. So which cluster you were meeting me for?  We were talking outside, contacting the target Kubernetes cluster.  Okay. That's running on the host. The server is running on the host network and then talking  to the, okay. Yeah. I thought you meant the services in the  Kubernetes cluster. It's kind of confusing. So there are no other questions. I have one.  So how do you authenticate? Like how do you, the Kubernetes cluster and the enterprise cluster  authenticate is just via HTTPS certificates? Although you have something additional on  top of that? No, just HTTPS certificates. So basically,  the authentication between the proxy and the client application is via HTTPS certificates,  of course. And then you have to specify some CA that you trust for the Kubernetes cluster.  And so the certificates for the Kubernetes cluster endpoint has to be signed by that  CA. So like in regular Kubernetes. Yeah.  Thank you for the talk. Thank you for the talk. So you have this thing where you do  the reverse proxy and you basically don't accept on a listening socket. And how do you  make sure that there's a connection ready when you need it?  Sorry. How do you make sure that there's a connection  ready to be accepted when you need to dial out essentially?  Yeah. So it's basically just a loop that listens on a port. And if it gets a connection there,  then yeah, it knows with some headers, it knows that it's from the wormhole connector.  And then it does this logic of doing the reverse thing. I'm not sure if that makes sense.  So it's just a very simple implementation. We didn't do anything fancy.  Cool. So thanks. And yeah, enjoy the conference.  Thanks, Jago.  Thank you.
HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_0_155.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_156_251.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_252_1199.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_1200_3755.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_3756_3839.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_3840_3911.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_3912_4727.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_4728_6455.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_6456_7931.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_7932_10535.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_10536_10655.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_10656_11003.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_11004_11075.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_11076_11123.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_11124_11231.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_11232_11279.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_11280_11327.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_11328_11915.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_11916_12491.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_12492_20423.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_20424_20519.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_20520_22631.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_22632_25415.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_25416_26039.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_26040_26387.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_26388_26819.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_26820_27539.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_27540_27815.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_27816_28043.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_28044_28259.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_28260_28307.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_28308_28595.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_28596_28955.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_28956_31127.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_31128_31979.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_31980_32075.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_32076_34547.png HTTP_tunneling_in_Go_using_HTTP_2_streams_u_JfIyYlLM4_34548_38698.png
Yeah. So I'm Jago. Hi. I'm a tech lead and co-founder at Kimfolk. I guess you've heard about Kimfolk, being that you are at this conference and there's a logo everywhere. But yeah. Basically we are a software consultancy. We're a team of software developers that help companies to work on their open source projects when they don't have resources or they don't have the expertise needed. But recently we have a bit of a shift. So we forked CoreOS container Linux and we called it Flatcar. And we are offering support for that. And soon we'll be announcing some other product related to Kubernetes. So stay tuned. Anyway. The plan for this talk is as follows. I will first introduce a problem that we were trying to solve. Then I will give a high level description of the program that we developed to solve this problem. I will talk a bit about the technologies used, surf raft and HTTP2. And then I will share some go re-implementation details that I found interesting. I will try to do a demo. Hopefully it will work. And then I will talk about what's missing and give some conclusion. So the problem is basically there's this enterprise systems that have a lot of legacy code and they're running in big monoliths or something like that. And they want to communicate with a Kima cluster. So Kima is this product by SAP. So it basically gives developers ways to easily deploy their applications in a Kubernetes cluster. And it gives some kind of serverless framework by default so they can use lambda functions and stuff like that. But for the purpose of this talk, just think of a regular Kubernetes cluster because I actually haven't even run the Kima cluster. I'm just dealing with playing Kubernetes. So they want to connect that cluster with some external services and they want to do it in a transparent way. So they want to be transparent to client applications and they want to reduce network latency between the cluster and the external system. And they also want to provide high availability and resilience in a simple way. So you can just deploy your application to this external system and you don't need any Kubernetes or any fancy distributed system. They just need something simple that can provide high availability. So yeah, I will give now a high level description of the wormhole connector, which is what we call this project. So basically it's just a distributed proxy. So being distributed, it provides high availability and if a node goes down, it can continue fine. It's based on using surf and raft, which are two distributed algorithms, which I will talk about a bit more later. And yeah, it connects the enterprise systems to Kima and Kima to enterprise systems. And it uses, to do that, it uses an HTTP tunnel between the enterprise system and Kima. So there's two components of this. One is the wormhole connector proper that runs on the enterprise solution side. And it's just a basic HTTP proxy. So services running on enterprise solution can talk to the Kima cluster by just setting the proxy variables there. So all the connections received from the enterprise solution services that will go through this tunnel to the Kubernetes cluster. And you can find a project there. The other component is the one that runs on the actual Kubernetes cluster. And what it does is it proxies the received connection from the tunnel to the services inside the cluster. But it also does the other way around. So it's also an HTTP proxy. So services inside the cluster can also talk to the enterprise system. And yeah, this is, yeah, it has some small example implementation there inside the same repository. So I have to say that this project was more like a proof of concept. So don't expect like a finalized software. So yeah, just a heads up. So I want to talk a bit about the technologies involved, surf and raft first. So yeah, surf is a Go-Sip distributed protocol. And yeah, it basically allows you to have some cluster membership and to pass some messages to all the members of your cluster. So you can see there in this GIF I found on the internet that some node receives some kind of message and it goes to the other nodes because one node talks to another node and then the other node talks to others. So in the end, the whole cluster will know this message that is being sent. Yeah. So we use this in the wormhole connector to basically add new members. So basically you just need to start the wormhole connector and tell it, okay, here's another member and then it will join the cluster. And yeah, it also provides a way to know when a member has disconnected to remove it from the cluster. And then we use raft, which is a consensus algorithm. It's used by things like etsyd console. And basically with raft, you have some certain state that you want to have in your cluster and you want to have it consistent. So if you query other members of the cluster, they have the same view of the state. And yeah, it provides a leader election. So you can have a leader that is doing all the right operations and then the others are followers who can, you know, you can read from them and get the same state. So in wormhole connector, we use that to elect a leader, of course. And we start some state in the raft SSM, which is finite state machine. And yeah, for now, it's just a simple array, but this has some application in this context because there's this concept of event bus events. And yeah, basically this legacy system can send events to the cluster. And we have to save the state there somewhere in the cluster. So we can have retries and we can have, yeah, things like that. In the future, you will start those events I was talking about, but for now it's just a simple array of strings because as I said, it's a proof of concept. Yeah, so if you want to learn more about drafts in a more detailed way, there's a lot of good resources on the internet. There's this page that has some nice visualizations and explanations. And of course, you can also read the raft paper, which is pretty good. Yeah, so as I said, I just implemented a simple queue. And you just send an event to one of the members here, connector one, for example. And then you can read it from another member and you will see that the same event is there. You can also delete the events. And yeah, that's basically the toy API that we've implemented. So talking about some other technology, HTTP 2. Basically, this is just the main differences between HTTP 1 and HTTP 2. So HTTP 2 provides header compression, so things are more efficient. And it's a binary format, so it's also a way to improve efficiency. And the most important part for us was that it has request response multiplexing. That means that with a single TCP connection, you can have several connections, HTTP connections, and they don't need several TCP connections like with HTTP 1. Yeah, so it's something like this. You have some HTTP 2 connection, which is a TCP connection between the client and the server. And then you have different streams. So this can be totally different data. And it will reuse the TCP connection, so you'll have to do the handshake again or the HTTP as negotiation. It's just a tunnel that stays there. So I will talk now about some implementation details of this. Yeah, first of all, the libraries I've used are the HashiCorp ones. And they were pretty cool because they were simple to use. Basically, for example, for SERV, you just have some default config. You configure some values like the bind address you're listening on and stuff like that. And you just create a SERV instance. And then you just get the events when a member joins a cluster or leaves a cluster. So that was pretty cool. And for RAV, same thing. So you need some kind of store on disk. We've chosen the bold DB store because of simplicity. And you just create a new store, create a snapshot store which can be backed by a file, for example. You need to choose a transport which shows the simple TCP. And then you need to give this finite state machine implementation. I'll talk about that in the next slide. And then you just start the RAV object. And yeah, that's pretty much it. So it's pretty easy to use. And yeah, well-documented. So this FSM thing, oops, sorry. This FSM interface is just some abstraction that the RAV library has. So you can have a finite state machine of your own data structure. So in this example, I just have this Go structure which has some slides of events which are just simple strings. And then some mutex to avoid problems with concurrency. And then you have to implement these operations. So yeah, you can change this structure and store whatever you want. And this state will be replicated through all the members of your cluster. So that was pretty neat, too. One thing to mention here is that the write operations are only, you can only do it on the leader. And that simplifies things for the protocol implementation. So what we do basically is just, if there's a write request, we redirect to the leader and then the request will be written. So this is the basic architecture of, like the full picture of this thing I was talking about, this project. So you see here that there's a wormhole connector and it has several boxes. So it's this distributed part. So the part on the left is this enterprise system. And the part on the right is the Kima cluster or Kubernetes. So basically you see here two connections, one in red on top. That's between the client and some Kubernetes service inside the cluster. And basically the client just sets a proxy to connect through the wormhole connector. The wormhole connector will have a tunnel to the wormhole dispatcher running in the cluster. And then the wormhole dispatcher will do the dispatching of the traffic to the services inside the Kubernetes cluster. You also have the other way around. So you have a Kubernetes spot there in blue, I guess, that is also configured with a proxy to the wormhole dispatcher. And then the connection will be routed through the tunnel again back to the connector and connected to a server in this enterprise system. So yeah, that's pretty much the architecture diagram. Yeah, so some other details to be able to do this of only using one TCP connection. Golang has a very simple way of doing this. So you just create your HTTP client. This HTTP client contains some transport. And as long as you use the same HTTP client, your requests will be routed through this one connection. So this is a pretty neat API and it's easy to use. And yeah, it just works. We didn't have to care much about how. And then tunnels, yay. So this is some chunk of code on the slide. So I guess it's not so nice. But I can try to go through it. So basically we have here two things. One is if the connection going through the proxy is HTTPS, it uses this connect method for HTTP and creates a tunnel to the other endpoint. And this is so because we don't want to get the unencrypted information in the proxy. So we need to do a tunnel. On the other case, it's just a simple HTTP without encryption. So the proxy will just remove some headers and will just pass the connection to the other side and then we'll read the tunnels. And then, yeah, that's pretty much it. For HTTPS, we create this tunnel. And this tunnel will use the same client. So it will reuse the connection for the tunnel. And then, yeah, we have some details there. If it's HTTP one, you have to hijack the connection. If it's HTTP two, you cannot do that because if you hijack the whole connection, the other streams on the connection will die. So you do something else. But yeah, you get the idea. It was pretty easy. One other interesting thing we're doing is if I go back to the diagram, yeah, there's a connection from the dispatcher to the connector. But we don't want the dispatcher to know the address of the connector. So what we did here is we did some kind of reverse tunnel. So the connector will connect to the dispatcher and then the roles will be inverted and then the client will be the dispatcher and the server will be the connector. So this is on this way in Go. So basically, the dispatcher will accept some TCP connection. You can create a transport with that. And then you can tell the transfer to return the connection that was established before to be the transport. And then you can create a client and then you can just do get or whatever. On the other side, you basically, this is the other side. So the connector will dial to the dispatcher. And once you have this connection, you can just set up a new server and tell it to serve the connection on this TCP connection. So basically, you do the reverse tunnel. Right. So now I'm going to try to do some demos. So let's see if they work. Let me change my screen configuration. Hopefully I don't break any video. Maybe I did. Can you see anything, Daniel? Okay. Okay. Well, there's some. Yeah, there we go. All right. So, well, let's start with this other demo. So I'm going to start three instances of the wormhole connector and they're going to be running in rocket containers so I can have, yeah, just because I developed a lot of on rocket and it's just easier for me. Yeah, I guess this is a very long command, but this, and it doesn't really render properly. But this basically sets up a bunch of options, starts the wormhole connector process. And yeah, that's pretty much it. And now you can see that here it says the node is a follower and then it gets promoted to a leader because there's only one member in the cluster. So I will start some, a couple more telling it the address of one of the members. So yeah, now we have here two followers in the bottom and one leader here. So now I was just, I'm just going to show this API that I mentioned of a simple distributed queue. So I can just, I don't know, post some event, event one to the first, the leader. And yeah, that works fine. I can just get it from the same leader. I get here event one. Or I can get it from another node. It's the same result. And yeah, I can just post again another one, like event two. I can post it to, I don't know, another node, like three. And but for this I need to enable redirection, so dash L. And now if I get an event from any, I posted the same, I don't know. Yeah. So it still has the old event because I haven't deleted it yet. So let's do that. And now if we query again, we see event two. We can delete that too. And yeah, that's the same. So that's one demo of this FSM thing. Now I want to show another demo of the whole workflow. So I have a mini queue Kubernetes cluster running. And yeah, you can see that it's running this wormhole dispatcher service I was mentioning. So now I can start the wormhole connector. Just, yeah, I set up some entries in my Etsy hosts. So it's certificate work. But basically I'm connecting to the dispatcher. And now you see this connected to wormhole dispatcher. So this means that the reverse connection was established. And now what I'm going to do is, yeah, I have here an echo server service inside the Kubernetes cluster. And this just returns the some information about a request that you made. So it's just a simple ping pong kind of thing. But this is running on the Kubernetes cluster. So in theory I shouldn't be able to access it on my host. But if I set the proxy to be the connector wormhole.io, this is an address pointing to this wormhole connector and some certificates so I can trust the CA. Then I can just use this URL, which is the name of a Kubernetes service with this port. And I'll just get a response. You can see here that there's an outgoing HTTP connection. And this goes to the cluster and it connects to this service inside. Yeah. So I can do this several times. And then if I do net stat, you still see only two connections going to that Kubernetes cluster. One is the reverse tunnel and the other one is the other tunnel. So the other thing I can do is just run some pod. Okay. So I'm running an alpine pod. I'm passing the CA here in an m variable. So the thing is trusted. So hopefully this will start. It doesn't want to start. Still creating. I guess I'm downloading this thing on the network. Oh, yeah. And this is a nice bug about QCTL of my version at least. That if you have already an image of the same name running, I mean, a deployment of the same name running, it just crashes. That's neat. Okay. This for some reason doesn't work. But what this demo will show is that you can get to a pod inside the Kubernetes cluster, configure the proxy to be the dispatcher, and then it will connect to my host. I was going to show it by just running net cat here. But this doesn't seem to work. Oh, it's running. So hopefully I can attach to it. Good. So, yeah. Let me pull some instructions here because I don't want to... Yeah. Okay. Internet is not really working well. Yeah. Anyway. Trust me that this works. That's it. Okay. So the missing pieces here, yeah. One thing that you notice is that there was two TCP connections, one on one side and one on the other side. I would think that theoretically it's possible to use just one TCP connection and that was the goal in the beginning. But I didn't manage to do it in the time that we had to work on this project. Yeah. I'm not sure if with the Go API you can do this or not or you have to do some other weird stuff. So if you have any ideas about this, please let me know. Yeah. And the other thing that's missing is this events support that I was mentioning. So basically the enterprise solution will send some event to the cluster and we will be able to store the events and try to deliver them to the cluster. And if that fails, we can do retries and all that stuff. So that's not implemented. And even though this is a simple program that you can run to create a cluster on the enterprise side, we also want to support deploying that in Kubernetes. And for that, of course, you don't need any server for RAF because Kubernetes already provides the distributed primitives that you need. There's some working progress implementation, but yeah, we didn't have much time. So yeah, in conclusion, yeah. So we've implemented two clients of distributed system libraries, two proxies, two HTTP tunnels, a lot of two. We learned a lot of these things because this is not something we've worked on before. And we had a lot of fun. So yeah, I hope, yeah, you enjoy this. Thanks. And yeah. Do we have any questions? Yeah. So we have about five minutes for questions. I assume the proxy pod has to run in host networking. Sorry. No, they don't. They just connect to this Kubernetes service and they will get to that dispatcher thing. And yeah, that will get routed. And then connections into the cluster. So from the external cluster into the Kubernetes cluster. Okay. So which cluster you were meeting me for? We were talking outside, contacting the target Kubernetes cluster. Okay. That's running on the host. The server is running on the host network and then talking to the, okay. Yeah. I thought you meant the services in the Kubernetes cluster. It's kind of confusing. So there are no other questions. I have one. So how do you authenticate? Like how do you, the Kubernetes cluster and the enterprise cluster authenticate is just via HTTPS certificates? Although you have something additional on top of that? No, just HTTPS certificates. So basically, the authentication between the proxy and the client application is via HTTPS certificates, of course. And then you have to specify some CA that you trust for the Kubernetes cluster. And so the certificates for the Kubernetes cluster endpoint has to be signed by that CA. So like in regular Kubernetes. Yeah. Thank you for the talk. Thank you for the talk. So you have this thing where you do the reverse proxy and you basically don't accept on a listening socket. And how do you make sure that there's a connection ready when you need it? Sorry. How do you make sure that there's a connection ready to be accepted when you need to dial out essentially? Yeah. So it's basically just a loop that listens on a port. And if it gets a connection there, then yeah, it knows with some headers, it knows that it's from the wormhole connector. And then it does this logic of doing the reverse thing. I'm not sure if that makes sense. So it's just a very simple implementation. We didn't do anything fancy. Cool. So thanks. And yeah, enjoy the conference. Thanks, Jago. Thank you.
technologies used, surf raft and HTTP2. And then the external system. And they also want to basically it's just a distributed proxy. So being based on using surf and raft, which are two enterprise systems to Kima and Kima to enterprise there. The other component is the one that runs cluster can also talk to the enterprise system. the cluster. And then we use raft, which is a several connections, HTTP connections, and they the Kima cluster or Kubernetes. So basically you will do the dispatching of the traffic to the One is if the connection going through the proxy the enterprise side, we also want to support the Kubernetes cluster. It's kind of confusing. the authentication between the proxy and the accept on a listening socket. And how do you make