Unity Game Engine
Sample, Hack
Surpur, Chetan
Chetan Surpur demonstrates a project he created that does anomaly detection on coordinates streaming out of the Unity Game Engine.
 Hi everyone, I'm Chaden. I'm going to show you guys a platform, so Unity is a game engine where you can create physical simulations, it's a game, a physics and graphics engine. You can create a physics simulation of some kind of simulation in a virtual environment. What I'll show you today is an API into Unity that allows you to get your data coming out of Unity and into NuPik, so you can do anomaly detection on virtual environments that you've created. This demo is mostly about how to get the data out of Unity into NuPik and then getting anomalies out of NuPik and into Unity. I haven't finally tuned the anomaly detection model, so that's just a caveat, but the actual data coming in and out works. The way it works is you can have a NuPik running in Python and you can stream data out of Unity into a Python server which NuPik reads from. Then it sends it back out into Unity. You run this Python client, which is a NuPik model that can get data in and return anomaly scores. Here's a simulation, here's a car driving around. What it'll do is it's going to send its global position, like a GPS position, in this virtual world and using the geospatial encoder to do anomaly detection on its GPS coordinates over time. I'll hit play here. I can't really lower the volume. As you drive around here, I'm controlling it right now, driving the car. You see that the trace that it leaves behind is red, so since this is not learned yet, it's a new pattern. Everything is anomalous. As I make this loop a couple of times, you'll see some greens. That's NuPik making better predictions as it learns the patterns and the anomaly scores are going down. If I do this a couple of times, eventually everything becomes pretty much green. Even though all the positions are slightly different, I'm not ever going to the exact same position each time. NuPik is able to generalize on that position using the geospatial encoder and learn these patterns. If I do something funky, so it's all green now. I can learn the path pretty well. If I just drive away somewhere else, it becomes red. It's detecting that that's anomalous behavior. If I sort of, okay, let me crash there. So now if I go back onto the path, NuPik recognizes it and it's not anomalous anymore. I can even drive sort of the opposite direction. Since it's a temporal sequence, it's detecting parts of that as anomalous. So anyway, that's the anomaly detection. Basically, this GPS, there's a sensor on this car. There's a sensor that is sending its data to this Python proxy server. NuPik is basically making request, HB request to that proxy server to get the GPS coordinates. Feeds it into a model, returns an anomaly score to the proxy server, which would send it to Unity. So there's a plugin here at this GitHub repository. There's instructions on how, sorry, you can get, you just import Unity plugin, and then you just insert lines of code into your Unity application to send data to NuPik and read data out of NuPik. And then you just run the proxy server and you can get the data out of NuPik. So here's the API with the Unity package you can import. And then all this code, the code for this demo I just showed you is here at this GitHub repo. And you can use that as a starting point for if you want to do a hack using Unity. Because it has a NuPik model running in it. So yeah, that's available there on GitHub. Yes? So can you learn multiple patterns? Yeah, so that's the question about the geospatial anomaly demo. And so the question is can it learn multiple patterns or multiple routes? And yes it can. So you can, I could have drove around horizontally instead of this vertical loop and it can recognize that as a pattern that sees anything outside of those two would be anomalous. So it can definitely learn multiple patterns. And there's more, I mean if you have more questions, there's a whole, we did a geospatial anomaly detection demo previously. And the code for that is also online, I think under the new Menta umbrella in GitHub. So you can check that out as well. Is the unlearning process as quick as the learning process? Sorry, can you repeat the question? Is the unlearning process for Unity or unlearning patterns? We have a capacity limit, but it's very, in the temporal memory, but it's very high. And so it's not really going to forget old stuff that easily. It does, you know, it filters out the signal from the noise. So that's why it takes some time to learn a sequence. You have to do it multiple times so it can find the consistencies in the pattern. But to forget things that it has strongly learned takes a lot more time than it takes to learn in the first place, which is what you want. Any other questions? Yes. Can you do multiple cars? Multiple cars. In your demo? Yeah, I tried to control the person. Right, right. So you could, I didn't, but you could have multiple ones. And the interface for this Unity API project is just, it doesn't really care what the inputs and outputs are. You can define them in your Unity project. You can have multiple coordinates coming off of multiple cars going into a new pick, or into the Python side. And then you can have multiple new pick models doing anomaly detection on each one of those coordinates. So you could set that up. There's no problem with that. Yes, any other questions? Okay, cool. So this code is here, and feel free to use it if you're interested in making a hack with Unity. I just posted it up here. It's here. Great. So, yeah. Give me a minute to set to Mike Austin, and he's going to talk about Project Tango. Thanks, Jaden. Thank you.
 Hi everyone, I'm Chaden. I'm going to show you guys a platform, so Unity is a game engine  where you can create physical simulations, it's a game, a physics and graphics engine.  You can create a physics simulation of some kind of simulation in a virtual environment.  What I'll show you today is an API into Unity that allows you to get your data coming out of Unity  and into NuPik, so you can do anomaly detection on virtual environments that you've created.  This demo is mostly about how to get the data out of Unity into NuPik and then getting anomalies out of NuPik and into Unity.  I haven't finally tuned the anomaly detection model, so that's just a caveat, but the actual data coming in and out works.  The way it works is you can have a NuPik running in Python and you can stream data out of Unity into a Python server which NuPik reads from.  Then it sends it back out into Unity.  You run this Python client, which is a NuPik model that can get data in and return anomaly scores.  Here's a simulation, here's a car driving around.  What it'll do is it's going to send its global position, like a GPS position, in this virtual world  and using the geospatial encoder to do anomaly detection on its GPS coordinates over time.  I'll hit play here. I can't really lower the volume.  As you drive around here, I'm controlling it right now, driving the car.  You see that the trace that it leaves behind is red, so since this is not learned yet, it's a new pattern.  Everything is anomalous.  As I make this loop a couple of times, you'll see some greens.  That's NuPik making better predictions as it learns the patterns and the anomaly scores are going down.  If I do this a couple of times, eventually everything becomes pretty much green.  Even though all the positions are slightly different, I'm not ever going to the exact same position each time.  NuPik is able to generalize on that position using the geospatial encoder and learn these patterns.  If I do something funky, so it's all green now.  I can learn the path pretty well. If I just drive away somewhere else, it becomes red.  It's detecting that that's anomalous behavior.  If I sort of, okay, let me crash there.  So now if I go back onto the path, NuPik recognizes it and it's not anomalous anymore.  I can even drive sort of the opposite direction.  Since it's a temporal sequence, it's detecting parts of that as anomalous.  So anyway, that's the anomaly detection.  Basically, this GPS, there's a sensor on this car.  There's a sensor that is sending its data to this Python proxy server.  NuPik is basically making request, HB request to that proxy server to get the GPS coordinates.  Feeds it into a model, returns an anomaly score to the proxy server, which would send it to Unity.  So there's a plugin here at this GitHub repository.  There's instructions on how, sorry, you can get, you just import Unity plugin,  and then you just insert lines of code into your Unity application to send data to NuPik and read data out of NuPik.  And then you just run the proxy server and you can get the data out of NuPik.  So here's the API with the Unity package you can import.  And then all this code, the code for this demo I just showed you is here at this GitHub repo.  And you can use that as a starting point for if you want to do a hack using Unity.  Because it has a NuPik model running in it.  So yeah, that's available there on GitHub.  Yes?  So can you learn multiple patterns?  Yeah, so that's the question about the geospatial anomaly demo.  And so the question is can it learn multiple patterns or multiple routes?  And yes it can.  So you can, I could have drove around horizontally instead of this vertical loop and it can recognize that as a pattern that sees anything outside of those two would be anomalous.  So it can definitely learn multiple patterns.  And there's more, I mean if you have more questions, there's a whole, we did a geospatial anomaly detection demo previously.  And the code for that is also online, I think under the new Menta umbrella in GitHub.  So you can check that out as well.  Is the unlearning process as quick as the learning process?  Sorry, can you repeat the question?  Is the unlearning process for Unity or unlearning patterns?  We have a capacity limit, but it's very, in the temporal memory, but it's very high.  And so it's not really going to forget old stuff that easily.  It does, you know, it filters out the signal from the noise.  So that's why it takes some time to learn a sequence.  You have to do it multiple times so it can find the consistencies in the pattern.  But to forget things that it has strongly learned takes a lot more time than it takes to learn in the first place, which is what you want.  Any other questions?  Yes.  Can you do multiple cars?  Multiple cars.  In your demo?  Yeah, I tried to control the person.  Right, right.  So you could, I didn't, but you could have multiple ones.  And the interface for this Unity API project is just, it doesn't really care what the inputs and outputs are.  You can define them in your Unity project.  You can have multiple coordinates coming off of multiple cars going into a new pick, or into the Python side.  And then you can have multiple new pick models doing anomaly detection on each one of those coordinates.  So you could set that up.  There's no problem with that.  Yes, any other questions?  Okay, cool.  So this code is here, and feel free to use it if you're interested in making a hack with Unity.  I just posted it up here.  It's here.  Great.  So, yeah.  Give me a minute to set to Mike Austin, and he's going to talk about Project Tango.  Thanks, Jaden.  Thank you.
Unity_Game_Engine_Hack_Sample__2015_Spring_NuPIC_Hackathon__0_251.png Unity_Game_Engine_Hack_Sample__2015_Spring_NuPIC_Hackathon__252_1055.png Unity_Game_Engine_Hack_Sample__2015_Spring_NuPIC_Hackathon__1056_1259.png Unity_Game_Engine_Hack_Sample__2015_Spring_NuPIC_Hackathon__1260_6347.png Unity_Game_Engine_Hack_Sample__2015_Spring_NuPIC_Hackathon__6348_6467.png Unity_Game_Engine_Hack_Sample__2015_Spring_NuPIC_Hackathon__6468_11891.png Unity_Game_Engine_Hack_Sample__2015_Spring_NuPIC_Hackathon__11892_11939.png Unity_Game_Engine_Hack_Sample__2015_Spring_NuPIC_Hackathon__11940_12263.png Unity_Game_Engine_Hack_Sample__2015_Spring_NuPIC_Hackathon__12264_12269.png
Hi everyone, I'm Chaden. I'm going to show you guys a platform, so Unity is a game engine where you can create physical simulations, it's a game, a physics and graphics engine. You can create a physics simulation of some kind of simulation in a virtual environment. What I'll show you today is an API into Unity that allows you to get your data coming out of Unity and into NuPik, so you can do anomaly detection on virtual environments that you've created. This demo is mostly about how to get the data out of Unity into NuPik and then getting anomalies out of NuPik and into Unity. I haven't finally tuned the anomaly detection model, so that's just a caveat, but the actual data coming in and out works. The way it works is you can have a NuPik running in Python and you can stream data out of Unity into a Python server which NuPik reads from. Then it sends it back out into Unity. You run this Python client, which is a NuPik model that can get data in and return anomaly scores. Here's a simulation, here's a car driving around. What it'll do is it's going to send its global position, like a GPS position, in this virtual world and using the geospatial encoder to do anomaly detection on its GPS coordinates over time. I'll hit play here. I can't really lower the volume. As you drive around here, I'm controlling it right now, driving the car. You see that the trace that it leaves behind is red, so since this is not learned yet, it's a new pattern. Everything is anomalous. As I make this loop a couple of times, you'll see some greens. That's NuPik making better predictions as it learns the patterns and the anomaly scores are going down. If I do this a couple of times, eventually everything becomes pretty much green. Even though all the positions are slightly different, I'm not ever going to the exact same position each time. NuPik is able to generalize on that position using the geospatial encoder and learn these patterns. If I do something funky, so it's all green now. I can learn the path pretty well. If I just drive away somewhere else, it becomes red. It's detecting that that's anomalous behavior. If I sort of, okay, let me crash there. So now if I go back onto the path, NuPik recognizes it and it's not anomalous anymore. I can even drive sort of the opposite direction. Since it's a temporal sequence, it's detecting parts of that as anomalous. So anyway, that's the anomaly detection. Basically, this GPS, there's a sensor on this car. There's a sensor that is sending its data to this Python proxy server. NuPik is basically making request, HB request to that proxy server to get the GPS coordinates. Feeds it into a model, returns an anomaly score to the proxy server, which would send it to Unity. So there's a plugin here at this GitHub repository. There's instructions on how, sorry, you can get, you just import Unity plugin, and then you just insert lines of code into your Unity application to send data to NuPik and read data out of NuPik. And then you just run the proxy server and you can get the data out of NuPik. So here's the API with the Unity package you can import. And then all this code, the code for this demo I just showed you is here at this GitHub repo. And you can use that as a starting point for if you want to do a hack using Unity. Because it has a NuPik model running in it. So yeah, that's available there on GitHub. Yes? So can you learn multiple patterns? Yeah, so that's the question about the geospatial anomaly demo. And so the question is can it learn multiple patterns or multiple routes? And yes it can. So you can, I could have drove around horizontally instead of this vertical loop and it can recognize that as a pattern that sees anything outside of those two would be anomalous. So it can definitely learn multiple patterns. And there's more, I mean if you have more questions, there's a whole, we did a geospatial anomaly detection demo previously. And the code for that is also online, I think under the new Menta umbrella in GitHub. So you can check that out as well. Is the unlearning process as quick as the learning process? Sorry, can you repeat the question? Is the unlearning process for Unity or unlearning patterns? We have a capacity limit, but it's very, in the temporal memory, but it's very high. And so it's not really going to forget old stuff that easily. It does, you know, it filters out the signal from the noise. So that's why it takes some time to learn a sequence. You have to do it multiple times so it can find the consistencies in the pattern. But to forget things that it has strongly learned takes a lot more time than it takes to learn in the first place, which is what you want. Any other questions? Yes. Can you do multiple cars? Multiple cars. In your demo? Yeah, I tried to control the person. Right, right. So you could, I didn't, but you could have multiple ones. And the interface for this Unity API project is just, it doesn't really care what the inputs and outputs are. You can define them in your Unity project. You can have multiple coordinates coming off of multiple cars going into a new pick, or into the Python side. And then you can have multiple new pick models doing anomaly detection on each one of those coordinates. So you could set that up. There's no problem with that. Yes, any other questions? Okay, cool. So this code is here, and feel free to use it if you're interested in making a hack with Unity. I just posted it up here. It's here. Great. So, yeah. Give me a minute to set to Mike Austin, and he's going to talk about Project Tango. Thanks, Jaden. Thank you.
and you can stream data out of Unity into a anomaly detection on its GPS coordinates over that proxy server to get the GPS coordinates.