Community ID Flow Hashing
Kreibich, Christian
Network security practitioners frequently need to correlate logs and alerts produced by the systems installed in their networks. For example, a Suricata alert might require the context of Zeek's connection logs for the alert to become actionable. Normally the best way to make such correlations is by manually identifying the flow tuple involved, in each of the monitor outputs involved, around the timestamps in question - a tedious and error-prone task. To simplify this process we're standardizing a straightforward algorithm, dubbed "Community ID" (-id-spec), that produces short textual hashes that reliably identify network flows directly at the source. Flow correlation then becomes a straightforward string comparison operation. Popular open-source network monitoring solutions now include support for this emerging standard, including Suricata, Wireshark, and Zeek, and there's a growing library of reusable implementations in various common programming languages.
 Welcome everybody. Thank you for joining my talk. My name is Christian Krivik. I'm an engineer at CoreLite where I'm part of the open source team and work full-time on the Zeke network monitor. Let's get started and let me share my screen. So today I'm going to be talking about Community ID, which is a standardized approach for flow hashing across your network security monitoring tool chain. Let me spend a couple of minutes motivating the problem. So if you've ever used Siricata, you might be familiar with the kinds of logs it produces. And here on the left hand side, we have a snippet of its eVlog that captures an alert that it flagged. And on the right hand side, we have a bit of HTTP logging that it is also capable of producing. And if you want to later on correlate some of the activity in those logs, it's really easy because Siricata flags every flow that it encounters with a unique flow identifier. In Siricata's case, it looks numeric and you can see in those two snippets that the identifier is the same. So you know it's referring to the same kind of flow. So that makes correlation really easy and that is cool. If you move on to Zeke, which you might know from its logs, on the left hand side here, we have a connection log and on the right hand side, we have an HTTP log. Then the same holds. Zeke also puts a unique identifier for every connection, for every flow it encounters in the traffic into its logs, making correlation really simple. So here the actual identifier looks a little differently because Zeke computes it differently. But the usability is the same. You can easily correlate across those flows. And that is also cool. Now, if you start working with both of those systems and if you're a practitioner, you might know that that is a very common setting, then you'll notice that you have a problem because those identifiers are computed differently and so they no longer match. And so as a workaround, you would basically have to identify, in each of those logs, how exactly a flow tuple is expressed, including all of the various little idiosyncrasies, and then basically abstract from that and come up with your own way for implementing the comparison and make sure that you don't care about directionality and so forth and identify things the same way. And that is really not so great to work with. So I think you can see where I'm going with this. We have good news for you. There is a better way to do this and it is called community ID. So community ID is a standardized algorithm, really simple for hashing flow tuples in multiple different network security monitoring applications. Let me spend a couple of minutes to explain how that works. It is intentionally really simple, almost embarrassingly simple. The result of that algorithm is just a string. It consists initially of a version string. Right now there is only a version one, so that part is only ever the number one followed by a colon followed by the actual computed hash. And that takes two parts as input. The first is a seed. The seed value is there so that in case you're facing multiple networks and you want to ensure that community ID values that you compute in both of those settings can never collide, then you can use different seed values in those two settings and rule out that possibility. Then the next part of the input is just a five tuple that is constructed from what you would expect, namely the actual source destination IP port and transport protocol. And if you look at the spec on our website, it spells out in quite a bit of detail how exactly you need to serialize, render that five tuple into the hashing algorithm. The hashing algorithm itself is SHA-1, which is really sort of the most basic thing available. We did that intentionally. It was the simplest thing to use that is in lots of crypto libraries, standard libraries, and so forth. So really one of the most accessible options. And in order to compress down the rendered hash result, we just run it through base 64 to make sure that the string itself becomes a little bit more manageable because it's just a little bit shorter. So that is just sort of a visual compression, if you will. And that's really it. So this is incredibly straightforward. And let me show you how that looks in practice. So we're going back to the same log snippets that I showed earlier. You have the Z-log on the left and the Surya catalog on the right with those flow tuples that the systems natively produce, but that don't quite match. And now if you enable community support in both of those systems, then you get an extra entry in those logs, community ID, and you see immediately that those strings match. And so the correlation task is trivial. It's just a string search and comparison. And you know, you're now looking at entries for the same flow. And that is super awesome. All right. So let me expand on a couple of the standard use cases. Some of I've already sort of touched on, but the most common one is probably really that you have multiple different applications installed in your network that are all producing logs where you would like to simplify for flow correlation. And you can do that by basically enabling community ID so that those strings get produced automatically. You can also use it for same correlation. So if your sim ingests all those logs, then the correlation task is really much simpler because now in your sim, you just need to do string comparison, which is easily supported. Another application that is sort of perhaps a little obvious, but that is also worth flagging, is if you have multiple installations of the same system in different places, then you can intentionally configure them such so that community string computations always match. You could potentially use the native algorithms that the systems provide for this as well. But if you get used to using community ID, then in a way you're more sort of future proof because if you then embrace its use in other applications, then you don't need to change your implementation. You can just continue to use the same string comparisons. There are other sort of more funky applications. The next one here was flagged to us by folks at ESNet. Thanks. That was actually pretty cool. They face a lot of asymmetric routing. So they see one direction of a flow go in one place and the other direction in another place in their network. But since the community ID algorithm is not sensitive to directionality of the traffic, the resulting hash values are the same. And so you can still use community ID in those cases to identify the fact that the same flow was present in those two locations. That's kind of funky and not something that we anticipated originally. And then of course, you are not required to apply community ID at the point of log generation. You don't have to use that in Siricata and Zeke and so forth. You can create those strings in post-production, if you will. You could ingest the logs and then create those strings for future comparison purposes, which is just as feasible. There's nothing that requires that algorithm to run as traffic is observed. So these are just a couple of examples that I hope you find useful. The current status. So the spec, reference implementations, reference data, if you want to come up with your own implementations and verify that the algorithms are correct. And just a snapshot of the current status of support across different systems and languages is always on our GitHub page. This is just github.com, correlate community ID spec. And let me highlight a couple of things. So in terms of language support right now, if you have a system that is written in C in Golang, Java, Python, Ruby, then adding support is really easy because there are reusable libraries or packages available for those languages that you can just drop in and start using it. And in particular for Python, for example, there's also a command line client that you can plug a flow tuple into and obtain the community ID string. So you're not forced to see that in traffic. First, you can just also synthesize those strings and verify them that way. So pretty good basis at this point for a lot of systems. I'll flag a couple of sort of the behemoths in the ecosystem. Surikada has supported community ID since 4.1 is one of the first to support it. Wireshark supports it as well, though that is a little bit more recent. This is as a version 3.4 or if you use development builds, I think it was in 3.2, if I remember correctly. And for Zeke, there's a package for its package manager, ZKG, that is compatible with any Zeke version as of 2.5 and newer and that is really basically covering anything that you should realistically be using at the moment. That is not all. If you go to that GitHub page, you'll see what other sort of systems out there have added support. This is a snapshot that I think is complete as of today. Thank you to anybody who's been involved in adding support to the system. It's really heartening. It is really exciting to see. And you can see that, you know, like if you're at all invested in any of these systems, you can really sort of just like flip the switch and start using community ID and it will pretty much be supported across a lot of your infrastructure in open source. Now, I might you say, no, no, no, this is so simple. This is almost stupid. You forgot so many things. What about, right, let me address a couple of the concerns you might have. So first of all, yes, exactly. This thing is not feature complete. That V1 was very intentionally sort of targeting just ease of implementation and getting a bit of feedback. There's a whole bunch of stuff that we might add for a version two. And one that I think would be really nice to have is basically an ability to make that algorithm configurable in terms of its inputs and outputs. So for example, in this example shown here, like you might want to say, well, in this particular installation, we have included VLANs Q and Q or other sort of like link layer headers. As part of the hashing algorithm, we did not use SHA1. We used SHA256 or we just ditched base encoding altogether. And one idea for this would be that in addition to that version number at the very beginning of the string, we also include like really short sort of basically config strings that just via a couple of letters capture how the algorithm was configured so that you can see that in the string itself and ensure that you're comparing apples to apples basically. Another thing that often is mentioned is performance versus collisions. So clearly, there is a potential here for things to be too slow, say, like you might say that, okay, SHA1 is clearly not one of the fastest hashing algorithms out there you might want to use like murmur two or what have you. And that is true. That might well be true, but it was not clear to us from the outset that performance is such a critical concern that we adopt something sort of less commonly known just to squeeze out extra performance. Collisions, likewise, it gets clear that for two flows that we're seeing sort of far apart in time, or that we're actually present in different parts of the network were semantically actually different flows, you would get collisions. Again, not quite clear that it is big enough a problem to warrant implementing something more complex. We've so far not really heard feedback that this has been prohibitive for anybody. If that is you, if you do find this is barely usable because of such concerns, then let us know and we can reflect that in future implementations. And I just mentioned time, and that is sort of a fun one because intuitively, many people say like, well, but the time is not in that hash. So if I encounter the same flow tuple, you know, an hour apart, then the hash will be the same. And indeed, it would be right now. And we were kind of treading carefully here because if you do relatively straightforward approaches, like, you know, you round the timestamp to some interval, like an hour, a minute, whatever it might be, then you also introduce a risk that different monitors compute that hash value a little bit differently because, you know, the clock in one is just before the end of that window, the other one is just after that window, you get different results and so forth. So I think this is sort of an interesting area for a clever algorithm that incorporates time, but sort of in a robust fashion. And if you have ideas for how to do this, we've honestly not spent too much time thinking about this, then let's hear it. That would be a really welcome contribution. Other basing codings is sort of one that is dear to my heart because I feel that the choice of base 64 was sort of, you know, like again, sort of the obvious one, but actually not that great in that the output is sometimes a little ugly, you know, base 64 and those strings, you know, have lots of sort of funky characters in them. And we've heard some reports that at least in some scenes, you have to sort of go out of your way because some of those characters are considered special and therefore make the use of that string sort of more difficult than it needs to be. And so the obvious contenders would be like base 62 or base 58, even depending on like how many sort of characters you want to rule out. I personally think that base 62 would be sufficient and sort of addresses that problem. But basically, this is another tension sort of length versus sort of possibility. And finally, there is sort of anonymity versus reversibility. It is fact that right now the produced community ID strings are not reversible. You cannot go back to the original flow tuple just knowing the hash. And, you know, if you still sit on the logs that include the community ID, you can locate it and then look up what that flow was. But you cannot do that just based on the string itself. Now, for a lot of use cases, that is perfectly sufficient because you are only ever going to look for string comparison or because you, you know, want to ask your buddies, hey, okay, this flow was nasty, it delivered some payload. Have you seen this too? But it is completely true that in other settings, you might want this to be reversible. And that is not something that is currently addressed by community ID. Like you could maybe just come up with a standard for writing out a flow tuple in a way that is not sensitive to directionality or some such. But it's sort of out of scope right now. All right, so with that, I'm done. That's all I had. Thank you so much. Again, if you want to check out the spec, the status of current implementations or just send some feedback, create some bug reports, whatever it might be, then go to our GitHub page. We welcome all of your feedback. If you are tempted to add support to an existing system or support for a new language, let us know too. Like it's always good to tell people ahead of time so that you don't waste time and effort. But thank you so much. Thank you so much to everybody who has already contributed to it or added it to some system. I really look forward to your questions and I'll see you in a couple minutes. Thank you, guys.
 Welcome everybody. Thank you for joining my talk. My name is Christian Krivik. I'm an engineer at  CoreLite where I'm part of the open source team and work full-time on the Zeke network monitor.  Let's get started and let me share my screen. So today I'm going to be talking about Community ID,  which is a standardized approach for flow hashing across your network security monitoring tool chain.  Let me spend a couple of minutes motivating the problem. So if you've ever used Siricata,  you might be familiar with the kinds of logs it produces. And here on the left hand side,  we have a snippet of its eVlog that captures an alert that it flagged. And on the right hand side,  we have a bit of HTTP logging that it is also capable of producing. And if you want to later  on correlate some of the activity in those logs, it's really easy because Siricata flags every flow  that it encounters with a unique flow identifier. In Siricata's case, it looks numeric and you  can see in those two snippets that the identifier is the same. So you know it's referring to the  same kind of flow. So that makes correlation really easy and that is cool. If you move on to Zeke,  which you might know from its logs, on the left hand side here, we have a connection log and on  the right hand side, we have an HTTP log. Then the same holds. Zeke also puts a unique identifier  for every connection, for every flow it encounters in the traffic into its logs, making correlation  really simple. So here the actual identifier looks a little differently because Zeke computes it  differently. But the usability is the same. You can easily correlate across those flows. And  that is also cool. Now, if you start working with both of those systems and if you're a practitioner,  you might know that that is a very common setting, then you'll notice that you have a problem because  those identifiers are computed differently and so they no longer match. And so as a workaround,  you would basically have to identify, in each of those logs, how exactly a flow tuple is expressed,  including all of the various little idiosyncrasies, and then basically abstract from that and come  up with your own way for implementing the comparison and make sure that you don't care about directionality  and so forth and identify things the same way. And that is really not so great to work with.  So I think you can see where I'm going with this. We have good news for you. There is a better way  to do this and it is called community ID. So community ID is a standardized algorithm,  really simple for hashing flow tuples in multiple different network security monitoring applications.  Let me spend a couple of minutes to explain how that works. It is intentionally really simple,  almost embarrassingly simple. The result of that algorithm is just a string. It consists initially  of a version string. Right now there is only a version one, so that part is only ever the number  one followed by a colon followed by the actual computed hash. And that takes two parts as input.  The first is a seed. The seed value is there so that in case you're facing multiple networks and  you want to ensure that community ID values that you compute in both of those settings  can never collide, then you can use different seed values in those two settings and rule out  that possibility. Then the next part of the input is just a five tuple that is constructed from  what you would expect, namely the actual source destination IP port and transport protocol.  And if you look at the spec on our website, it spells out in quite a bit of detail how exactly  you need to serialize, render that five tuple into the hashing algorithm. The hashing algorithm  itself is SHA-1, which is really sort of the most basic thing available. We did that intentionally.  It was the simplest thing to use that is in lots of crypto libraries, standard libraries,  and so forth. So really one of the most accessible options. And in order to compress down the  rendered hash result, we just run it through base 64 to make sure that the string itself becomes a  little bit more manageable because it's just a little bit shorter. So that is just sort of a  visual compression, if you will. And that's really it. So this is incredibly straightforward.  And let me show you how that looks in practice. So we're going back to the same log snippets that  I showed earlier. You have the Z-log on the left and the Surya catalog on the right with those  flow tuples that the systems natively produce, but that don't quite match. And now if you enable  community support in both of those systems, then you get an extra entry in those logs, community ID,  and you see immediately that those strings match. And so the correlation task is trivial. It's just  a string search and comparison. And you know, you're now looking at entries for the same flow.  And that is super awesome. All right. So let me expand on a couple of the standard use cases.  Some of I've already sort of touched on, but the most common one is probably really that you have  multiple different applications installed in your network that are all producing logs where you  would like to simplify for flow correlation. And you can do that by basically enabling community  ID so that those strings get produced automatically. You can also use it for same correlation. So if  your sim ingests all those logs, then the correlation task is really much simpler because  now in your sim, you just need to do string comparison, which is easily supported.  Another application that is sort of perhaps a little obvious, but that is also worth flagging,  is if you have multiple installations of the same system in different places, then you can  intentionally configure them such so that community string computations always match.  You could potentially use the native algorithms that the systems provide for this as well. But if  you get used to using community ID, then in a way you're more sort of future proof because if you then  embrace its use in other applications, then you don't need to change your implementation. You can  just continue to use the same string comparisons. There are other sort of more funky applications.  The next one here was flagged to us by folks at ESNet. Thanks. That was actually pretty cool.  They face a lot of asymmetric routing. So they see one direction of a flow go in one place and the  other direction in another place in their network. But since the community ID algorithm is not  sensitive to directionality of the traffic, the resulting hash values are the same.  And so you can still use community ID in those cases to identify the fact that the same flow  was present in those two locations. That's kind of funky and not something that we anticipated  originally. And then of course, you are not required to apply community ID at the point of log generation.  You don't have to use that in Siricata and Zeke and so forth. You can create those strings in  post-production, if you will. You could ingest the logs and then create those strings for future  comparison purposes, which is just as feasible. There's nothing that requires that algorithm to run  as traffic is observed. So these are just a couple of examples that I hope you find useful.  The current status. So the spec, reference implementations, reference data, if you want to  come up with your own implementations and verify that the algorithms are correct. And just a snapshot  of the current status of support across different systems and languages is always on our GitHub  page. This is just github.com, correlate community ID spec. And let me highlight a couple of things.  So in terms of language support right now, if you have a system that is written in C in Golang,  Java, Python, Ruby, then adding support is really easy because there are reusable libraries or  packages available for those languages that you can just drop in and start using it. And in particular  for Python, for example, there's also a command line client that you can plug a flow tuple into and  obtain the community ID string. So you're not forced to see that in traffic. First, you can just  also synthesize those strings and verify them that way. So pretty good basis at this point for a lot  of systems. I'll flag a couple of sort of the behemoths in the ecosystem. Surikada has supported  community ID since 4.1 is one of the first to support it. Wireshark supports it as well,  though that is a little bit more recent. This is as a version 3.4 or if you use development builds,  I think it was in 3.2, if I remember correctly. And for Zeke, there's a package for its package  manager, ZKG, that is compatible with any Zeke version as of 2.5 and newer and that is really  basically covering anything that you should realistically be using at the moment. That is  not all. If you go to that GitHub page, you'll see what other sort of systems out there have added  support. This is a snapshot that I think is complete as of today. Thank you to anybody who's  been involved in adding support to the system. It's really heartening. It is really exciting to see.  And you can see that, you know, like if you're at all invested in any of these systems, you can  really sort of just like flip the switch and start using community ID and it will pretty much be  supported across a lot of your infrastructure in open source. Now, I might you say, no, no, no,  this is so simple. This is almost stupid. You forgot so many things. What about, right, let me  address a couple of the concerns you might have. So first of all, yes, exactly. This thing is not  feature complete. That V1 was very intentionally sort of targeting just ease of implementation  and getting a bit of feedback. There's a whole bunch of stuff that we might add for a version two.  And one that I think would be really nice to have is basically an ability to make that algorithm  configurable in terms of its inputs and outputs. So for example, in this example shown here,  like you might want to say, well, in this particular installation, we have included VLANs Q and Q  or other sort of like link layer headers. As part of the hashing algorithm, we did not use  SHA1. We used SHA256 or we just ditched base encoding altogether. And one idea for this would  be that in addition to that version number at the very beginning of the string, we also include  like really short sort of basically config strings that just via a couple of letters capture how the  algorithm was configured so that you can see that in the string itself and ensure that you're  comparing apples to apples basically. Another thing that often is mentioned is performance  versus collisions. So clearly, there is a potential here for things to be too slow, say,  like you might say that, okay, SHA1 is clearly not one of the fastest hashing algorithms out  there you might want to use like murmur two or what have you. And that is true. That might well  be true, but it was not clear to us from the outset that performance is such a critical concern that  we adopt something sort of less commonly known just to squeeze out extra performance. Collisions,  likewise, it gets clear that for two flows that we're seeing sort of far apart in time,  or that we're actually present in different parts of the network were semantically actually  different flows, you would get collisions. Again, not quite clear that it is big enough  a problem to warrant implementing something more complex. We've so far not really heard feedback  that this has been prohibitive for anybody. If that is you, if you do find this is barely usable  because of such concerns, then let us know and we can reflect that in future implementations.  And I just mentioned time, and that is sort of a fun one because intuitively, many people say like,  well, but the time is not in that hash. So if I encounter the same flow tuple, you know, an hour  apart, then the hash will be the same. And indeed, it would be right now. And we were kind of treading  carefully here because if you do relatively straightforward approaches, like, you know,  you round the timestamp to some interval, like an hour, a minute, whatever it might be, then you  also introduce a risk that different monitors compute that hash value a little bit differently  because, you know, the clock in one is just before the end of that window, the other one is just  after that window, you get different results and so forth. So I think this is sort of  an interesting area for a clever algorithm that incorporates time, but sort of in a robust fashion.  And if you have ideas for how to do this, we've honestly not spent too much time thinking about  this, then let's hear it. That would be a really welcome contribution. Other basing codings is  sort of one that is dear to my heart because I feel that the choice of base 64 was sort of, you  know, like again, sort of the obvious one, but actually not that great in that the output is  sometimes a little ugly, you know, base 64 and those strings, you know, have lots of sort of  funky characters in them. And we've heard some reports that at least in some scenes,  you have to sort of go out of your way because some of those characters are considered special  and therefore make the use of that string sort of more difficult than it needs to be.  And so the obvious contenders would be like base 62 or base 58, even depending on like how many  sort of characters you want to rule out. I personally think that base 62 would be sufficient  and sort of addresses that problem. But basically, this is another tension sort of length versus  sort of possibility. And finally, there is sort of anonymity versus reversibility. It is fact  that right now the produced community ID strings are not reversible. You cannot go back to the  original flow tuple just knowing the hash. And, you know, if you still sit on the logs that  include the community ID, you can locate it and then look up what that flow was. But you cannot  do that just based on the string itself. Now, for a lot of use cases, that is perfectly sufficient  because you are only ever going to look for string comparison or because you, you know,  want to ask your buddies, hey, okay, this flow was nasty, it delivered some payload.  Have you seen this too? But it is completely true that in other settings, you might want this to be  reversible. And that is not something that is currently addressed by community ID. Like you  could maybe just come up with a standard for writing out a flow tuple in a way that is not  sensitive to directionality or some such. But it's sort of out of scope right now.  All right, so with that, I'm done. That's all I had. Thank you so much. Again, if you want to  check out the spec, the status of current implementations or just send some feedback, create  some bug reports, whatever it might be, then go to our GitHub page. We welcome all of your feedback.  If you are tempted to add support to an existing system or support for a new language, let us know  too. Like it's always good to tell people ahead of time so that you don't waste time and effort.  But thank you so much. Thank you so much to everybody who has already contributed to it or  added it to some system. I really look forward to your questions and I'll see you in a couple minutes.  Thank you, guys.
sdn_community_id_0_179.png sdn_community_id_180_323.png sdn_community_id_324_491.png sdn_community_id_492_743.png sdn_community_id_744_851.png sdn_community_id_852_1895.png sdn_community_id_1896_2855.png sdn_community_id_2856_3863.png sdn_community_id_3864_3971.png sdn_community_id_3972_4403.png sdn_community_id_4404_7043.png sdn_community_id_7044_7847.png sdn_community_id_7848_7931.png sdn_community_id_7932_11375.png sdn_community_id_11376_13715.png sdn_community_id_13716_14531.png sdn_community_id_14532_14579.png sdn_community_id_14580_14759.png sdn_community_id_14760_16451.png sdn_community_id_16452_18179.png sdn_community_id_18180_19619.png sdn_community_id_19620_20855.png sdn_community_id_20856_22367.png sdn_community_id_22368_23291.png sdn_community_id_23292_23686.png
Welcome everybody. Thank you for joining my talk. My name is Christian Krivik. I'm an engineer at CoreLite where I'm part of the open source team and work full-time on the Zeke network monitor. Let's get started and let me share my screen. So today I'm going to be talking about Community ID, which is a standardized approach for flow hashing across your network security monitoring tool chain. Let me spend a couple of minutes motivating the problem. So if you've ever used Siricata, you might be familiar with the kinds of logs it produces. And here on the left hand side, we have a snippet of its eVlog that captures an alert that it flagged. And on the right hand side, we have a bit of HTTP logging that it is also capable of producing. And if you want to later on correlate some of the activity in those logs, it's really easy because Siricata flags every flow that it encounters with a unique flow identifier. In Siricata's case, it looks numeric and you can see in those two snippets that the identifier is the same. So you know it's referring to the same kind of flow. So that makes correlation really easy and that is cool. If you move on to Zeke, which you might know from its logs, on the left hand side here, we have a connection log and on the right hand side, we have an HTTP log. Then the same holds. Zeke also puts a unique identifier for every connection, for every flow it encounters in the traffic into its logs, making correlation really simple. So here the actual identifier looks a little differently because Zeke computes it differently. But the usability is the same. You can easily correlate across those flows. And that is also cool. Now, if you start working with both of those systems and if you're a practitioner, you might know that that is a very common setting, then you'll notice that you have a problem because those identifiers are computed differently and so they no longer match. And so as a workaround, you would basically have to identify, in each of those logs, how exactly a flow tuple is expressed, including all of the various little idiosyncrasies, and then basically abstract from that and come up with your own way for implementing the comparison and make sure that you don't care about directionality and so forth and identify things the same way. And that is really not so great to work with. So I think you can see where I'm going with this. We have good news for you. There is a better way to do this and it is called community ID. So community ID is a standardized algorithm, really simple for hashing flow tuples in multiple different network security monitoring applications. Let me spend a couple of minutes to explain how that works. It is intentionally really simple, almost embarrassingly simple. The result of that algorithm is just a string. It consists initially of a version string. Right now there is only a version one, so that part is only ever the number one followed by a colon followed by the actual computed hash. And that takes two parts as input. The first is a seed. The seed value is there so that in case you're facing multiple networks and you want to ensure that community ID values that you compute in both of those settings can never collide, then you can use different seed values in those two settings and rule out that possibility. Then the next part of the input is just a five tuple that is constructed from what you would expect, namely the actual source destination IP port and transport protocol. And if you look at the spec on our website, it spells out in quite a bit of detail how exactly you need to serialize, render that five tuple into the hashing algorithm. The hashing algorithm itself is SHA-1, which is really sort of the most basic thing available. We did that intentionally. It was the simplest thing to use that is in lots of crypto libraries, standard libraries, and so forth. So really one of the most accessible options. And in order to compress down the rendered hash result, we just run it through base 64 to make sure that the string itself becomes a little bit more manageable because it's just a little bit shorter. So that is just sort of a visual compression, if you will. And that's really it. So this is incredibly straightforward. And let me show you how that looks in practice. So we're going back to the same log snippets that I showed earlier. You have the Z-log on the left and the Surya catalog on the right with those flow tuples that the systems natively produce, but that don't quite match. And now if you enable community support in both of those systems, then you get an extra entry in those logs, community ID, and you see immediately that those strings match. And so the correlation task is trivial. It's just a string search and comparison. And you know, you're now looking at entries for the same flow. And that is super awesome. All right. So let me expand on a couple of the standard use cases. Some of I've already sort of touched on, but the most common one is probably really that you have multiple different applications installed in your network that are all producing logs where you would like to simplify for flow correlation. And you can do that by basically enabling community ID so that those strings get produced automatically. You can also use it for same correlation. So if your sim ingests all those logs, then the correlation task is really much simpler because now in your sim, you just need to do string comparison, which is easily supported. Another application that is sort of perhaps a little obvious, but that is also worth flagging, is if you have multiple installations of the same system in different places, then you can intentionally configure them such so that community string computations always match. You could potentially use the native algorithms that the systems provide for this as well. But if you get used to using community ID, then in a way you're more sort of future proof because if you then embrace its use in other applications, then you don't need to change your implementation. You can just continue to use the same string comparisons. There are other sort of more funky applications. The next one here was flagged to us by folks at ESNet. Thanks. That was actually pretty cool. They face a lot of asymmetric routing. So they see one direction of a flow go in one place and the other direction in another place in their network. But since the community ID algorithm is not sensitive to directionality of the traffic, the resulting hash values are the same. And so you can still use community ID in those cases to identify the fact that the same flow was present in those two locations. That's kind of funky and not something that we anticipated originally. And then of course, you are not required to apply community ID at the point of log generation. You don't have to use that in Siricata and Zeke and so forth. You can create those strings in post-production, if you will. You could ingest the logs and then create those strings for future comparison purposes, which is just as feasible. There's nothing that requires that algorithm to run as traffic is observed. So these are just a couple of examples that I hope you find useful. The current status. So the spec, reference implementations, reference data, if you want to come up with your own implementations and verify that the algorithms are correct. And just a snapshot of the current status of support across different systems and languages is always on our GitHub page. This is just github.com, correlate community ID spec. And let me highlight a couple of things. So in terms of language support right now, if you have a system that is written in C in Golang, Java, Python, Ruby, then adding support is really easy because there are reusable libraries or packages available for those languages that you can just drop in and start using it. And in particular for Python, for example, there's also a command line client that you can plug a flow tuple into and obtain the community ID string. So you're not forced to see that in traffic. First, you can just also synthesize those strings and verify them that way. So pretty good basis at this point for a lot of systems. I'll flag a couple of sort of the behemoths in the ecosystem. Surikada has supported community ID since 4.1 is one of the first to support it. Wireshark supports it as well, though that is a little bit more recent. This is as a version 3.4 or if you use development builds, I think it was in 3.2, if I remember correctly. And for Zeke, there's a package for its package manager, ZKG, that is compatible with any Zeke version as of 2.5 and newer and that is really basically covering anything that you should realistically be using at the moment. That is not all. If you go to that GitHub page, you'll see what other sort of systems out there have added support. This is a snapshot that I think is complete as of today. Thank you to anybody who's been involved in adding support to the system. It's really heartening. It is really exciting to see. And you can see that, you know, like if you're at all invested in any of these systems, you can really sort of just like flip the switch and start using community ID and it will pretty much be supported across a lot of your infrastructure in open source. Now, I might you say, no, no, no, this is so simple. This is almost stupid. You forgot so many things. What about, right, let me address a couple of the concerns you might have. So first of all, yes, exactly. This thing is not feature complete. That V1 was very intentionally sort of targeting just ease of implementation and getting a bit of feedback. There's a whole bunch of stuff that we might add for a version two. And one that I think would be really nice to have is basically an ability to make that algorithm configurable in terms of its inputs and outputs. So for example, in this example shown here, like you might want to say, well, in this particular installation, we have included VLANs Q and Q or other sort of like link layer headers. As part of the hashing algorithm, we did not use SHA1. We used SHA256 or we just ditched base encoding altogether. And one idea for this would be that in addition to that version number at the very beginning of the string, we also include like really short sort of basically config strings that just via a couple of letters capture how the algorithm was configured so that you can see that in the string itself and ensure that you're comparing apples to apples basically. Another thing that often is mentioned is performance versus collisions. So clearly, there is a potential here for things to be too slow, say, like you might say that, okay, SHA1 is clearly not one of the fastest hashing algorithms out there you might want to use like murmur two or what have you. And that is true. That might well be true, but it was not clear to us from the outset that performance is such a critical concern that we adopt something sort of less commonly known just to squeeze out extra performance. Collisions, likewise, it gets clear that for two flows that we're seeing sort of far apart in time, or that we're actually present in different parts of the network were semantically actually different flows, you would get collisions. Again, not quite clear that it is big enough a problem to warrant implementing something more complex. We've so far not really heard feedback that this has been prohibitive for anybody. If that is you, if you do find this is barely usable because of such concerns, then let us know and we can reflect that in future implementations. And I just mentioned time, and that is sort of a fun one because intuitively, many people say like, well, but the time is not in that hash. So if I encounter the same flow tuple, you know, an hour apart, then the hash will be the same. And indeed, it would be right now. And we were kind of treading carefully here because if you do relatively straightforward approaches, like, you know, you round the timestamp to some interval, like an hour, a minute, whatever it might be, then you also introduce a risk that different monitors compute that hash value a little bit differently because, you know, the clock in one is just before the end of that window, the other one is just after that window, you get different results and so forth. So I think this is sort of an interesting area for a clever algorithm that incorporates time, but sort of in a robust fashion. And if you have ideas for how to do this, we've honestly not spent too much time thinking about this, then let's hear it. That would be a really welcome contribution. Other basing codings is sort of one that is dear to my heart because I feel that the choice of base 64 was sort of, you know, like again, sort of the obvious one, but actually not that great in that the output is sometimes a little ugly, you know, base 64 and those strings, you know, have lots of sort of funky characters in them. And we've heard some reports that at least in some scenes, you have to sort of go out of your way because some of those characters are considered special and therefore make the use of that string sort of more difficult than it needs to be. And so the obvious contenders would be like base 62 or base 58, even depending on like how many sort of characters you want to rule out. I personally think that base 62 would be sufficient and sort of addresses that problem. But basically, this is another tension sort of length versus sort of possibility. And finally, there is sort of anonymity versus reversibility. It is fact that right now the produced community ID strings are not reversible. You cannot go back to the original flow tuple just knowing the hash. And, you know, if you still sit on the logs that include the community ID, you can locate it and then look up what that flow was. But you cannot do that just based on the string itself. Now, for a lot of use cases, that is perfectly sufficient because you are only ever going to look for string comparison or because you, you know, want to ask your buddies, hey, okay, this flow was nasty, it delivered some payload. Have you seen this too? But it is completely true that in other settings, you might want this to be reversible. And that is not something that is currently addressed by community ID. Like you could maybe just come up with a standard for writing out a flow tuple in a way that is not sensitive to directionality or some such. But it's sort of out of scope right now. All right, so with that, I'm done. That's all I had. Thank you so much. Again, if you want to check out the spec, the status of current implementations or just send some feedback, create some bug reports, whatever it might be, then go to our GitHub page. We welcome all of your feedback. If you are tempted to add support to an existing system or support for a new language, let us know too. Like it's always good to tell people ahead of time so that you don't waste time and effort. But thank you so much. Thank you so much to everybody who has already contributed to it or added it to some system. I really look forward to your questions and I'll see you in a couple minutes. Thank you, guys.
across your network security monitoring tool correlate across those flows. And that is also might know that that is a very common setting, identify, in each of those logs, how exactly a flow tuple is expressed, including all of the one followed by a colon followed by the actual 64 to make sure that the string itself becomes a installed in your network that are all producing those logs, then the correlation task is really need to do string comparison, which is easily the systems provide for this as well. But if you place in their network. But since the community ID algorithm is not sensitive to directionality are not required to apply community ID at the first to support it. Wireshark supports it as remember correctly. And for Zeke, there's a Thank you to anybody who's been involved in terms of its inputs and outputs. So for example, string, we also include like really short sort of It is fact that right now the produced community flow tuple in a way that is not sensitive to