Keep appetite for the stats, it costs nothing
Kerhor, Arthur de
Bouattour, Hedi
VPP (aka Vector Packet Processing) is a fast network stack running in Linux userspace. It is designed to handle packets with high performance, which makes gathering statistics efficiently a must have. The model that has been chosen in VPP to provide up to date statistics is built upon shared memory and optimistic locking. The counters are updated in this shared memory at a rather low cost by the data plane and can be read out at almost any time by all the consumers. We will first describe this model in more details. The consumption of these stats may take various forms depending on the use case and the application needs. That's why we have developed different high-level components to access them: 1) A filesystem in userspace: thanks to go-fuse, we can mount a filesystem ordering statistics in folders and files, in a similar fashion to '/proc' in Linux 2) A Prometheus agent: applied to Calico/VPP, a new dataplane for Calico - the popular cloud native Kubernetes network plugin - based on VPP. Prometheus is integrated as a monitoring tool in order to export our statistics in the form of real-time metrics collected from targets. Metrics come from our pod interfaces statistics, and targets are Calico/VPP agents running on our Kubernetes cluster nodes. During the presentation, you will have a quick demo of these components.
 Hello everyone, my name is Hedy, I am a software engineer at Cisco and I'm glad to present to you with Arthur the statistics consumption model in VPP and some of its applications. Hello everyone, I'm Arthur, I work at Cisco and I'm part of the team developing the vector packet processing software aka VPP and we use it in different projects in Cisco. So I hope you have not eaten too much because you'll need some appetite for the statistics and we'll have a look at the stats segment in VPP and also at various clients for the statistics. So first a quick intro on VPP and I think some of you are already familiar with this software so I'm going to be quick on this one. So VPP is an open source networking data plane working in user space. It is highly optimized for performance, speed and scale and it is designed as a graph of nodes. So we take some vectors of packets as inputs. The packets go through different set of nodes depending on their contents and they can be dropped, can be sent to another interface and so on. So what is the statistics segment in VPP? It is a model based on shared memory. We use vectors and we make the exception that we have a single writer which is the main threads that can add new counters to the statistics segment or that can extend some existing counters and that we have also multiple readers. We also make the assumption that the number of writes is really low in comparison to the number of reads and that's why we have adopted the optimistic locking strategy. What is important in the stats segment is that reading is really, really cheap. Actually, it costs nothing because we don't stop the data plane at all when doing the reads. We just read the shared memory. However, this means that it costs a bit in the data plane because we have to update the statistic counters and that may induce some cache line misses. So let's have a look at the shared memory layout. We have a main header containing the directory vector. It also contains an error vectors and an epic counter and in progress Boolean for the optimistic locking part. The directory vector consists of a list of statistic entries. Each entry may have a given type. For example, the simplest type is just a scholar. We have the example here of the number of worker threads. The data is simply the value of the scholar. We can also have a vector. For example, the interface Tx is a vector indexed by threads and interfaces. The data is simply the pointer to the array. So we have here the number of packets and the number of bytes per interface per thread. Then we have a sibling type. This type contains two indexes. The first one is the index of the entry in the directory vector and then we have the interface index which is the index of the row in the array. Note that when doing some updates in the data plane, we have the pointer to this array stored in the data plane so that we can directly access it. Now, let's talk about the optimistic locking parts. First, taking the lock means setting the input rest Boolean. Releasing the lock means un-setting it and incrementing the APO counter. When we do a write, a test, we add a counter to the stat segment or we extend one. For example, if we add a new interface, we take the lock, we update the directory vector if needed so pointers may change. We update also the counters vectors and then we release the lock. What it means for readers is that they need to wait until in progress is un-set and they need to store the input value, they read what they want to read and at the end of the read, they need to check that the epoch has not changed and that the in progress is not set. If it's not true, it means that there has been a write during the read, so we have to redo the read and we try again. We have no concurrency issues in the stat segments because when adding or extending the counters, it is done under a barrier by the main threads and this means that the work-route threads cannot update the counters in the meantime. Now, let's present some different clients we have in VPP that use the stat segments. The simplest client we have is a simple executable that connects to the stats. It's to have info about the shared memory location. It allows to list or dump the statistics available and it takes a list of read as input and which returns the matching counters. So either a list or a dump of the counters depending on the comment. So it is really handy to have a quick access to the statistics. It is easy to launch. It is to launch, sorry. So it's pretty handy to have raw access to the statistics. Now we have also developed a fuse file system in Gulleng thanks to the GoFuse model. In this model, we have different type of nodes. We can define directory nodes and file nodes. So directory node contains the bus in our case and it keeps the trace of the epoch. Then we have also file nodes which correspond to the counters and in these nodes we store the counter index so that we just have to look at the given index in the directory vector file instead of having to fetch the pointer each time. So let's take an example. When you open a directory in the file system, we look at the current epoch. If the epoch has changed since the last update of this directory, we need to update all the subdirectories and files into the subdirectories so that we are sure that there were no more ads or deletions in the meantime. And if there were some files added, for example, we could still update the directory and add the given counter to the directory. It would be the same if we add some deletions, we would just delete the counter inside this directory. And note that these updates are done recursively so that the subdirectories are also updated. When we want to read a file this time, we just access the given counter in the directory vector and we get the value as output. So now let's have a quick demo of this. On the left hand side, we have an instance of VPP running and we can access the interfaces in VPP, in the CLI. So for now we have only the default interface local0 as expected because we have not run any traffic. Now we can try to install and start the start file system as simple as a make install and a sudo make start. Here we go. So now we can try to access the file stem directory. So it's named startsfsdir and we can Ls all what we have in that directory in order to fetch all the data from the directory vector and update the epoch counters. So now we can have access to the interfaces name and as expected there is only this local0 interface. Now let's try to add some other interfaces and run some traffic with a little script in the VPP CLI. And here we go. Let's try to access the interfaces name again and a bunch of new interfaces have appeared. So let's try to access the counters for the new interface pgo for example. So we go into the directory interfaces pgo which is a simulink directory and we have a bunch of counter files. So let's try to for example cat inside ip4 and we have the number of packets, ip4 packets that have gone through these interfaces. We can also access the files corresponding to the Rx and Tx packets and bytes and here we have all the info we wanted for the Tx and Rx packets and bytes index per threads for this interface. Now we can also access some statistics about the processing nodes of the VPP. So we need to go inside the nodes directory and then we need to go inside the for example ip4 input directory which is the directory for the ip4 input node and see what we have. So we have a bunch of other counters and we can access them. So for example we can access the number of vectors per thread and there we are. Then we can also access the number of clocks or calls in order to have the number of calls for this node in VPP and there we are. I'm going to present another way of consumption for our VPP stats which is the Prometheus exporter that gathers the statistics. Our use case this time is Calico VPP. So what is Calico VPP? Calico is an open source Kubernetes networking solution and network policy. It also supports other platforms. It manages networking between pods, nodes, VMs. The main goal of integrating VPP is to accelerate the networking of Kubernetes clusters that use Calico. So instead of using standard Linux networking pipeline with IP tables as a data plane, nodes run the VPP data plane and we provide faster networking to their pods without requiring any changes to the applications running in the pods. This is really meant to be transparent. So when running VPP we do not have any additional requirements compared to regular Calico. When we use the VPP data plane, the VPP instance is inserted between the host and the external network. So on startup the host interface is replaced by an ump link interface and a ton interface to keep communicating with the outside as normal. As for the pods, VPP creates a ton interface for every pod. So VPP handles interfaces and packets transmission and creates a ton interface for every pod. Thus, it is very useful to gather statistics about pod interfaces such as Rx packets or Tx packets or number of errors, etc. So this is a high level view of the agent that you get on each node when you deploy Calico VPP. The agent is the processor responsible for all the Calico specific configuration in VPP. On this container we have all the runtime configuration of VPP in the form of running servers. The CNI server, the routing manager, the services manager and the policies manager. They interact with the Kubernetes and Calico APIs to configure VPP. In order to export our VPP stats as metrics, we add the new component, Prometheus server. As part of the agent, Prometheus server knows the state of the CNI server and the created pods interfaces. And every interval of time, it fetches the real time statistics through the VPP API in Calico VPP, which accesses the stat segment chaired memory and gets the statistics needed. So among stats, we actually select ones related to pod interfaces. So let's take a deeper look at how this works. Here are the nodes of our Kubernetes cluster. Pods are created on nodes, VPP adds interfaces for these pods, and we'd like to collect those interfaces' stats. To expose Prometheus metrics in our application, we need to provide a slash metrics HTTP endpoint on every node that we call a node exporter. Every node exporter is running on the Calico VPP agent. So it accesses the stat segment chaired memory through the VPP API, converts stats into Prometheus metrics, then exposes them on the HTTP server. How Prometheus server needs to be configured so that it targets those endpoints. Prometheus uses an HTTP pool model in order to export our statistics in the form of real time metrics recorded in a time series database. So targets are the slash metrics HTTP servers running on Kubernetes cluster nodes, and metrics come from our VPP ton interfaces statistics. This the performance of the interfaces in our system is displayed as a nice graph. Now let's watch this demo to better understand the feature. So we have these nodes in our Kubernetes cluster, and here are our code HTTP node. Every node has an HTTP endpoint that provides pod interfaces statistics. For example, on this node, we have these metrics. For configuration, this is how Prometheus is configured. We can see here the node IPs and the port as targets. Prometheus is serving on the 1990 port where it has access to the cluster to collect metrics. So let's take a look at the graph. Here are the different metrics provided. Let's select TX bytes, for example, for transmitted bytes, execute, and then this is the graph. So in the last five minutes, values are at zero. Let's run a test to see what happens. In a few seconds, we have this flow of the traffic for our test. Thanks for watching, and any questions that you might have or will come up. Okay guys, thank you for your talk. Let's start with a few questions. So how to coordinate information known by BPP, in G, the interface names, with the indexes you get in the share memory counters? So we have a couple of counters that list all the interfaces names and nodes names that you can also access in order to have the correct indexes in the other counters, and you can just map the indexes you get in the names files to the indexes you have in all the files and you get the stats interface, for example. Okay, nice. In the KQPP use case, how do you coordinate the application logic known by the control pane with the interface indexes? So let me answer that. The information coming from VPP stat segment concerns all VPP interfaces and are defined by indexes, and the application logic known by the control pane knows which container is giving the information and for every interface which pot is concerned, and the correlation between both is simply done by the fact that they are running in the same process. So Prometheus server is a part of Calico VPP agent control pane, and it runs the Go application that accesses the VPP stats. So this correlation is important because it helps select pod interfaces among the available ones. Okay, thank you. In case of the time, I think more questions could be answered while typing a chat. I want to thank you guys for coming around and have this nice talk. If you've got any questions, type it in the network there from chat and if you want something out, we've got 20 or 40 seconds left I guess. Thank you. Thank you for listening and thank you all for joining to this talk. Thanks. Have a nice day in Fossum 2022. Bye. Bye. Bye. Bye.
 Hello everyone, my name is Hedy, I am a software engineer at Cisco and I'm glad to present  to you with Arthur the statistics consumption model in VPP and some of its applications.  Hello everyone, I'm Arthur, I work at Cisco and I'm part of the team developing the vector  packet processing software aka VPP and we use it in different projects in Cisco.  So I hope you have not eaten too much because you'll need some appetite for the statistics  and we'll have a look at the stats segment in VPP and also at various clients for the  statistics.  So first a quick intro on VPP and I think some of you are already familiar with this  software so I'm going to be quick on this one.  So VPP is an open source networking data plane working in user space.  It is highly optimized for performance, speed and scale and it is designed as a graph of  nodes.  So we take some vectors of packets as inputs.  The packets go through different set of nodes depending on their contents and they can be  dropped, can be sent to another interface and so on.  So what is the statistics segment in VPP?  It is a model based on shared memory.  We use vectors and we make the exception that we have a single writer which is the main  threads that can add new counters to the statistics segment or that can extend some  existing counters and that we have also multiple readers.  We also make the assumption that the number of writes is really low in comparison to the  number of reads and that's why we have adopted the optimistic locking strategy.  What is important in the stats segment is that reading is really, really cheap.  Actually, it costs nothing because we don't stop the data plane at all when doing the  reads.  We just read the shared memory.  However, this means that it costs a bit in the data plane because we have to update the  statistic counters and that may induce some cache line misses.  So let's have a look at the shared memory layout.  We have a main header containing the directory vector.  It also contains an error vectors and an epic counter and in progress Boolean for the optimistic  locking part.  The directory vector consists of a list of statistic entries.  Each entry may have a given type.  For example, the simplest type is just a scholar.  We have the example here of the number of worker threads.  The data is simply the value of the scholar.  We can also have a vector.  For example, the interface Tx is a vector indexed by threads and interfaces.  The data is simply the pointer to the array.  So we have here the number of packets and the number of bytes per interface per thread.  Then we have a sibling type.  This type contains two indexes.  The first one is the index of the entry in the directory vector and then we have the  interface index which is the index of the row in the array.  Note that when doing some updates in the data plane, we have the pointer to this array stored  in the data plane so that we can directly access it.  Now, let's talk about the optimistic locking parts.  First, taking the lock means setting the input rest Boolean.  Releasing the lock means un-setting it and incrementing the APO counter.  When we do a write, a test, we add a counter to the stat segment or we extend one.  For example, if we add a new interface, we take the lock, we update the directory vector  if needed so pointers may change.  We update also the counters vectors and then we release the lock.  What it means for readers is that they need to wait until in progress is un-set and they  need to store the input value, they read what they want to read and at the end of the read,  they need to check that the epoch has not changed and that the in progress is not set.  If it's not true, it means that there has been a write during the read, so we have to  redo the read and we try again.  We have no concurrency issues in the stat segments because when adding or extending the counters,  it is done under a barrier by the main threads and this means that the work-route threads  cannot update the counters in the meantime.  Now, let's present some different clients we have in VPP that use the stat segments.  The simplest client we have is a simple executable that connects to the stats.  It's to have info about the shared memory location.  It allows to list or dump the statistics available and it takes a list of read as input and which  returns the matching counters.  So either a list or a dump of the counters depending on the comment.  So it is really handy to have a quick access to the statistics.  It is easy to launch.  It is to launch, sorry.  So it's pretty handy to have raw access to the statistics.  Now we have also developed a fuse file system in Gulleng thanks to the GoFuse model.  In this model, we have different type of nodes.  We can define directory nodes and file nodes.  So directory node contains the bus in our case and it keeps the trace of the epoch.  Then we have also file nodes which correspond to the counters and in these nodes we store  the counter index so that we just have to look at the given index in the directory vector  file instead of having to fetch the pointer each time.  So let's take an example.  When you open a directory in the file system, we look at the current epoch.  If the epoch has changed since the last update of this directory, we need to update all the  subdirectories and files into the subdirectories so that we are sure that there were no more  ads or deletions in the meantime.  And if there were some files added, for example, we could still update the directory and add  the given counter to the directory.  It would be the same if we add some deletions, we would just delete the counter inside this  directory.  And note that these updates are done recursively so that the subdirectories are also updated.  When we want to read a file this time, we just access the given counter in the directory  vector and we get the value as output.  So now let's have a quick demo of this.  On the left hand side, we have an instance of VPP running and we can access the interfaces  in VPP, in the CLI.  So for now we have only the default interface local0 as expected because we have not run  any traffic.  Now we can try to install and start the start file system as simple as a make install and  a sudo make start.  Here we go.  So now we can try to access the file stem directory.  So it's named startsfsdir and we can Ls all what we have in that directory in order to  fetch all the data from the directory vector and update the epoch counters.  So now we can have access to the interfaces name and as expected there is only this local0  interface.  Now let's try to add some other interfaces and run some traffic with a little script  in the VPP CLI.  And here we go.  Let's try to access the interfaces name again and a bunch of new interfaces have appeared.  So let's try to access the counters for the new interface pgo for example.  So we go into the directory interfaces pgo which is a simulink directory and we have  a bunch of counter files.  So let's try to for example cat inside ip4 and we have the number of packets, ip4 packets  that have gone through these interfaces.  We can also access the files corresponding to the Rx and Tx packets and bytes and here  we have all the info we wanted for the Tx and Rx packets and bytes index per threads  for this interface.  Now we can also access some statistics about the processing nodes of the VPP.  So we need to go inside the nodes directory and then we need to go inside the for example  ip4 input directory which is the directory for the ip4 input node and see what we have.  So we have a bunch of other counters and we can access them.  So for example we can access the number of vectors per thread and there we are.  Then we can also access the number of clocks or calls in order to have the number of calls  for this node in VPP and there we are.  I'm going to present another way of consumption for our VPP stats which is the Prometheus exporter  that gathers the statistics.  Our use case this time is Calico VPP.  So what is Calico VPP?  Calico is an open source Kubernetes networking solution and network policy.  It also supports other platforms.  It manages networking between pods, nodes, VMs.  The main goal of integrating VPP is to accelerate the networking of Kubernetes clusters that  use Calico.  So instead of using standard Linux networking pipeline with IP tables as a data plane, nodes  run the VPP data plane and we provide faster networking to their pods without requiring  any changes to the applications running in the pods.  This is really meant to be transparent.  So when running VPP we do not have any additional requirements compared to regular Calico.  When we use the VPP data plane, the VPP instance is inserted between the host and the external  network.  So on startup the host interface is replaced by an ump link interface and a ton interface  to keep communicating with the outside as normal.  As for the pods, VPP creates a ton interface for every pod.  So VPP handles interfaces and packets transmission and creates a ton interface for every pod.  Thus, it is very useful to gather statistics about pod interfaces such as Rx packets or  Tx packets or number of errors, etc.  So this is a high level view of the agent that you get on each node when you deploy Calico  VPP.  The agent is the processor responsible for all the Calico specific configuration in VPP.  On this container we have all the runtime configuration of VPP in the form of running  servers.  The CNI server, the routing manager, the services manager and the policies manager.  They interact with the Kubernetes and Calico APIs to configure VPP.  In order to export our VPP stats as metrics, we add the new component, Prometheus server.  As part of the agent, Prometheus server knows the state of the CNI server and the created  pods interfaces.  And every interval of time, it fetches the real time statistics through the VPP API in  Calico VPP, which accesses the stat segment chaired memory and gets the statistics needed.  So among stats, we actually select ones related to pod interfaces.  So let's take a deeper look at how this works.  Here are the nodes of our Kubernetes cluster.  Pods are created on nodes, VPP adds interfaces for these pods, and we'd like to collect  those interfaces' stats.  To expose Prometheus metrics in our application, we need to provide a slash metrics HTTP endpoint  on every node that we call a node exporter.  Every node exporter is running on the Calico VPP agent.  So it accesses the stat segment chaired memory through the VPP API, converts stats into Prometheus  metrics, then exposes them on the HTTP server.  How Prometheus server needs to be configured so that it targets those endpoints.  Prometheus uses an HTTP pool model in order to export our statistics in the form of real  time metrics recorded in a time series database.  So targets are the slash metrics HTTP servers running on Kubernetes cluster nodes, and metrics  come from our VPP ton interfaces statistics.  This the performance of the interfaces in our system is displayed as a nice graph.  Now let's watch this demo to better understand the feature.  So we have these nodes in our Kubernetes cluster, and here are our code HTTP node.  Every node has an HTTP endpoint that provides pod interfaces statistics.  For example, on this node, we have these metrics.  For configuration, this is how Prometheus is configured.  We can see here the node IPs and the port as targets.  Prometheus is serving on the 1990 port where it has access to the cluster to collect metrics.  So let's take a look at the graph.  Here are the different metrics provided.  Let's select TX bytes, for example, for transmitted bytes, execute, and then this is the graph.  So in the last five minutes, values are at zero.  Let's run a test to see what happens.  In a few seconds, we have this flow of the traffic for our test.  Thanks for watching, and any questions that you might have or will come up.  Okay guys, thank you for your talk.  Let's start with a few questions.  So how to coordinate information known by BPP, in G, the interface names, with the indexes  you get in the share memory counters?  So we have a couple of counters that list all the interfaces names and nodes names that  you can also access in order to have the correct indexes in the other counters, and you can  just map the indexes you get in the names files to the indexes you have in all the files  and you get the stats interface, for example.  Okay, nice.  In the KQPP use case, how do you coordinate the application logic known by the control  pane with the interface indexes?  So let me answer that.  The information coming from VPP stat segment concerns all VPP interfaces and are defined  by indexes, and the application logic known by the control pane knows which container  is giving the information and for every interface which pot is concerned, and the correlation  between both is simply done by the fact that they are running in the same process.  So Prometheus server is a part of Calico VPP agent control pane, and it runs the Go application  that accesses the VPP stats.  So this correlation is important because it helps select pod interfaces among the available  ones.  Okay, thank you.  In case of the time, I think more questions could be answered while typing a chat.  I want to thank you guys for coming around and have this nice talk.  If you've got any questions, type it in the network there from chat and if you want something  out, we've got 20 or 40 seconds left I guess.  Thank you.  Thank you for listening and thank you all for joining to this talk.  Thanks.  Have a nice day in Fossum 2022.  Bye.  Bye.  Bye.  Bye.
keep_appetite_for_the_stats_0_155.png keep_appetite_for_the_stats_156_1247.png keep_appetite_for_the_stats_1248_3719.png keep_appetite_for_the_stats_3720_4847.png keep_appetite_for_the_stats_4848_5999.png keep_appetite_for_the_stats_6000_6383.png keep_appetite_for_the_stats_6384_18899.png keep_appetite_for_the_stats_18900_20291.png keep_appetite_for_the_stats_20292_20987.png keep_appetite_for_the_stats_20988_21995.png keep_appetite_for_the_stats_21996_22379.png keep_appetite_for_the_stats_22380_23615.png keep_appetite_for_the_stats_23616_23795.png keep_appetite_for_the_stats_23796_23915.png keep_appetite_for_the_stats_23916_24083.png keep_appetite_for_the_stats_24084_25295.png keep_appetite_for_the_stats_25296_26339.png keep_appetite_for_the_stats_26340_26531.png keep_appetite_for_the_stats_26532_27983.png keep_appetite_for_the_stats_27984_28211.png keep_appetite_for_the_stats_28212_28259.png keep_appetite_for_the_stats_28260_28427.png keep_appetite_for_the_stats_28428_28511.png keep_appetite_for_the_stats_28512_28633.png
Hello everyone, my name is Hedy, I am a software engineer at Cisco and I'm glad to present to you with Arthur the statistics consumption model in VPP and some of its applications. Hello everyone, I'm Arthur, I work at Cisco and I'm part of the team developing the vector packet processing software aka VPP and we use it in different projects in Cisco. So I hope you have not eaten too much because you'll need some appetite for the statistics and we'll have a look at the stats segment in VPP and also at various clients for the statistics. So first a quick intro on VPP and I think some of you are already familiar with this software so I'm going to be quick on this one. So VPP is an open source networking data plane working in user space. It is highly optimized for performance, speed and scale and it is designed as a graph of nodes. So we take some vectors of packets as inputs. The packets go through different set of nodes depending on their contents and they can be dropped, can be sent to another interface and so on. So what is the statistics segment in VPP? It is a model based on shared memory. We use vectors and we make the exception that we have a single writer which is the main threads that can add new counters to the statistics segment or that can extend some existing counters and that we have also multiple readers. We also make the assumption that the number of writes is really low in comparison to the number of reads and that's why we have adopted the optimistic locking strategy. What is important in the stats segment is that reading is really, really cheap. Actually, it costs nothing because we don't stop the data plane at all when doing the reads. We just read the shared memory. However, this means that it costs a bit in the data plane because we have to update the statistic counters and that may induce some cache line misses. So let's have a look at the shared memory layout. We have a main header containing the directory vector. It also contains an error vectors and an epic counter and in progress Boolean for the optimistic locking part. The directory vector consists of a list of statistic entries. Each entry may have a given type. For example, the simplest type is just a scholar. We have the example here of the number of worker threads. The data is simply the value of the scholar. We can also have a vector. For example, the interface Tx is a vector indexed by threads and interfaces. The data is simply the pointer to the array. So we have here the number of packets and the number of bytes per interface per thread. Then we have a sibling type. This type contains two indexes. The first one is the index of the entry in the directory vector and then we have the interface index which is the index of the row in the array. Note that when doing some updates in the data plane, we have the pointer to this array stored in the data plane so that we can directly access it. Now, let's talk about the optimistic locking parts. First, taking the lock means setting the input rest Boolean. Releasing the lock means un-setting it and incrementing the APO counter. When we do a write, a test, we add a counter to the stat segment or we extend one. For example, if we add a new interface, we take the lock, we update the directory vector if needed so pointers may change. We update also the counters vectors and then we release the lock. What it means for readers is that they need to wait until in progress is un-set and they need to store the input value, they read what they want to read and at the end of the read, they need to check that the epoch has not changed and that the in progress is not set. If it's not true, it means that there has been a write during the read, so we have to redo the read and we try again. We have no concurrency issues in the stat segments because when adding or extending the counters, it is done under a barrier by the main threads and this means that the work-route threads cannot update the counters in the meantime. Now, let's present some different clients we have in VPP that use the stat segments. The simplest client we have is a simple executable that connects to the stats. It's to have info about the shared memory location. It allows to list or dump the statistics available and it takes a list of read as input and which returns the matching counters. So either a list or a dump of the counters depending on the comment. So it is really handy to have a quick access to the statistics. It is easy to launch. It is to launch, sorry. So it's pretty handy to have raw access to the statistics. Now we have also developed a fuse file system in Gulleng thanks to the GoFuse model. In this model, we have different type of nodes. We can define directory nodes and file nodes. So directory node contains the bus in our case and it keeps the trace of the epoch. Then we have also file nodes which correspond to the counters and in these nodes we store the counter index so that we just have to look at the given index in the directory vector file instead of having to fetch the pointer each time. So let's take an example. When you open a directory in the file system, we look at the current epoch. If the epoch has changed since the last update of this directory, we need to update all the subdirectories and files into the subdirectories so that we are sure that there were no more ads or deletions in the meantime. And if there were some files added, for example, we could still update the directory and add the given counter to the directory. It would be the same if we add some deletions, we would just delete the counter inside this directory. And note that these updates are done recursively so that the subdirectories are also updated. When we want to read a file this time, we just access the given counter in the directory vector and we get the value as output. So now let's have a quick demo of this. On the left hand side, we have an instance of VPP running and we can access the interfaces in VPP, in the CLI. So for now we have only the default interface local0 as expected because we have not run any traffic. Now we can try to install and start the start file system as simple as a make install and a sudo make start. Here we go. So now we can try to access the file stem directory. So it's named startsfsdir and we can Ls all what we have in that directory in order to fetch all the data from the directory vector and update the epoch counters. So now we can have access to the interfaces name and as expected there is only this local0 interface. Now let's try to add some other interfaces and run some traffic with a little script in the VPP CLI. And here we go. Let's try to access the interfaces name again and a bunch of new interfaces have appeared. So let's try to access the counters for the new interface pgo for example. So we go into the directory interfaces pgo which is a simulink directory and we have a bunch of counter files. So let's try to for example cat inside ip4 and we have the number of packets, ip4 packets that have gone through these interfaces. We can also access the files corresponding to the Rx and Tx packets and bytes and here we have all the info we wanted for the Tx and Rx packets and bytes index per threads for this interface. Now we can also access some statistics about the processing nodes of the VPP. So we need to go inside the nodes directory and then we need to go inside the for example ip4 input directory which is the directory for the ip4 input node and see what we have. So we have a bunch of other counters and we can access them. So for example we can access the number of vectors per thread and there we are. Then we can also access the number of clocks or calls in order to have the number of calls for this node in VPP and there we are. I'm going to present another way of consumption for our VPP stats which is the Prometheus exporter that gathers the statistics. Our use case this time is Calico VPP. So what is Calico VPP? Calico is an open source Kubernetes networking solution and network policy. It also supports other platforms. It manages networking between pods, nodes, VMs. The main goal of integrating VPP is to accelerate the networking of Kubernetes clusters that use Calico. So instead of using standard Linux networking pipeline with IP tables as a data plane, nodes run the VPP data plane and we provide faster networking to their pods without requiring any changes to the applications running in the pods. This is really meant to be transparent. So when running VPP we do not have any additional requirements compared to regular Calico. When we use the VPP data plane, the VPP instance is inserted between the host and the external network. So on startup the host interface is replaced by an ump link interface and a ton interface to keep communicating with the outside as normal. As for the pods, VPP creates a ton interface for every pod. So VPP handles interfaces and packets transmission and creates a ton interface for every pod. Thus, it is very useful to gather statistics about pod interfaces such as Rx packets or Tx packets or number of errors, etc. So this is a high level view of the agent that you get on each node when you deploy Calico VPP. The agent is the processor responsible for all the Calico specific configuration in VPP. On this container we have all the runtime configuration of VPP in the form of running servers. The CNI server, the routing manager, the services manager and the policies manager. They interact with the Kubernetes and Calico APIs to configure VPP. In order to export our VPP stats as metrics, we add the new component, Prometheus server. As part of the agent, Prometheus server knows the state of the CNI server and the created pods interfaces. And every interval of time, it fetches the real time statistics through the VPP API in Calico VPP, which accesses the stat segment chaired memory and gets the statistics needed. So among stats, we actually select ones related to pod interfaces. So let's take a deeper look at how this works. Here are the nodes of our Kubernetes cluster. Pods are created on nodes, VPP adds interfaces for these pods, and we'd like to collect those interfaces' stats. To expose Prometheus metrics in our application, we need to provide a slash metrics HTTP endpoint on every node that we call a node exporter. Every node exporter is running on the Calico VPP agent. So it accesses the stat segment chaired memory through the VPP API, converts stats into Prometheus metrics, then exposes them on the HTTP server. How Prometheus server needs to be configured so that it targets those endpoints. Prometheus uses an HTTP pool model in order to export our statistics in the form of real time metrics recorded in a time series database. So targets are the slash metrics HTTP servers running on Kubernetes cluster nodes, and metrics come from our VPP ton interfaces statistics. This the performance of the interfaces in our system is displayed as a nice graph. Now let's watch this demo to better understand the feature. So we have these nodes in our Kubernetes cluster, and here are our code HTTP node. Every node has an HTTP endpoint that provides pod interfaces statistics. For example, on this node, we have these metrics. For configuration, this is how Prometheus is configured. We can see here the node IPs and the port as targets. Prometheus is serving on the 1990 port where it has access to the cluster to collect metrics. So let's take a look at the graph. Here are the different metrics provided. Let's select TX bytes, for example, for transmitted bytes, execute, and then this is the graph. So in the last five minutes, values are at zero. Let's run a test to see what happens. In a few seconds, we have this flow of the traffic for our test. Thanks for watching, and any questions that you might have or will come up. Okay guys, thank you for your talk. Let's start with a few questions. So how to coordinate information known by BPP, in G, the interface names, with the indexes you get in the share memory counters? So we have a couple of counters that list all the interfaces names and nodes names that you can also access in order to have the correct indexes in the other counters, and you can just map the indexes you get in the names files to the indexes you have in all the files and you get the stats interface, for example. Okay, nice. In the KQPP use case, how do you coordinate the application logic known by the control pane with the interface indexes? So let me answer that. The information coming from VPP stat segment concerns all VPP interfaces and are defined by indexes, and the application logic known by the control pane knows which container is giving the information and for every interface which pot is concerned, and the correlation between both is simply done by the fact that they are running in the same process. So Prometheus server is a part of Calico VPP agent control pane, and it runs the Go application that accesses the VPP stats. So this correlation is important because it helps select pod interfaces among the available ones. Okay, thank you. In case of the time, I think more questions could be answered while typing a chat. I want to thank you guys for coming around and have this nice talk. If you've got any questions, type it in the network there from chat and if you want something out, we've got 20 or 40 seconds left I guess. Thank you. Thank you for listening and thank you all for joining to this talk. Thanks. Have a nice day in Fossum 2022. Bye. Bye. Bye. Bye.
with Arthur the statistics consumption model in the statistics and we'll have a look at the stats the statistics. So first a quick intro on VPP and is designed as a graph of nodes. So we take some through different set of nodes depending on their statistics segment in VPP? It is a model based on shared memory. We use vectors and we make the the number of reads and that's why we have Boolean for the optimistic locking part. The in the data plane, we have the pointer to this that there has been a write during the read, so bunch of other counters and we can access them. statistics. Our use case this time is Calico VPP. provide a slash metrics HTTP endpoint on every exporter is running on the Calico VPP agent. So server. How Prometheus server needs to be Prometheus uses an HTTP pool model in order to export our statistics in the form of real time targets are the slash metrics HTTP servers running on Kubernetes cluster nodes, and metrics come from our VPP ton interfaces statistics. This have these nodes in our Kubernetes cluster, and HTTP endpoint that provides pod interfaces you have in all the files and you get the stats and the application logic known by the control