 If you ever find yourself in Pittsburgh and want to catch a glimpse of the future, Carnegie Mellon University is probably a good place to start. To get inside the robotics lab, you'll need to walk past Tank. He's a robo-ceptionist and, well, actually Tank is really important to this story, but he isn't what I want to show you right now. This is. Looking at this large geodesic dome covered in wires, you can't help but feel like you're staring into a time machine, and in a way you kind of are. But rather than traveling through time, this panoptic studio is capturing it. Using an array of cameras, researchers here are able to capture and then replicate any action into a 3D model. From a dance to a cello performance, it can perfectly reconstruct these moments in time. And while that's pretty impressive by itself, it's just the starting point in a quest to decipher an age-old code. So basically you can see that this is a kind of geodesic dome. Looks like a sphere. We specifically designed this architecture so that we can put cameras as uniform as possible. And you can see also that we specially modularized the system so that we only designed one panel and we can duplicate the panels. One panel is composed of 24 small cameras and one high-resolution cameras and they are saved in the local hard drives and in the end all the local data are they are transmitted to the NAS. And so that sounds like a ton of data, like how much data is in one minute of capture? So one minute of data is about 600 gigabytes. That's huge data. That's a big file. In fact, to this point, the Panoptix Studio has captured over one petabyte of data. That's about a million gigabytes. And the latest is, well, me. Ready? Can you see me? Yeah? So while I'm dancing, those 500 cameras are capturing every movement I make. And after a couple of weeks of processing, it looks something like this. This point cloud view is not only showing a bunch of tracking points, it's tracing the paths they could take while I'm moving around. All right, glad I stretched before that. Now, motion capture by itself isn't anything new. Similar technology has been used to bring digital characters to life in films for years now. Motion capture technology has gotten so good that an actor like Andy Serkis can really inject life into a fully digital character. But to do so requires a full motion capture suit with tracking markers placed all over your body, even painted on your face. Now, this works well enough for trying to bring a dragon to life, but it's hard to feel natural in that kind of equipment. The Panoptix Studio, in contrast, allows for similar tracking, but without any of the physical markers. You can just freely move your body in a way that feels natural and unrestricted, which is a necessity for them to capture what they are really looking for. A really good quote from Edward Sapir, who is a linguist. He says, there's an elaborate and secret code that is known to no one, but understood by all. And he's referring to all the gestures that we use to communicate. And that's kind of the code that we're trying to sort of crack or understand. Our goal, essentially, is to endow machines with the capacity to understand these social interactions. So that's where Tank comes back into play. You remember Tank, the roboceptionist? He can see me via camera and react to my presence, but he can't understand more than what I type to him. So when I start to get frustrated that he isn't helping me find the Panoptix Studio, he can't read that frustration or react in a way that helps ease my tension, which kind of makes him bad at his job. Machines have a very poor understanding of what all those things are. So, for example, if I'm not paying attention to you or if I'm looking at you and paying attention, those are the kinds of things that machines just don't respond to right now. And that's kind of a big deal. Studies have suggested that when we communicate with each other, more than half of the message we're sending is based on the way we move. Our posture, facial expressions, and gestures can at times convey more than our words alone. Here, let me show you. So now you can see me and you can read what my body is telling you, even if my words are saying something different. Maybe from my posture you can tell I'm in a cramped little sound booth, or you can tell I'm excited about this story, that I'm nervous about reading from a script, that I'm hopelessly improvising into a camera to try and give you a sense of how expressive body language can be. When we communicate with each other, we use many interesting hand gestures, which is really important to communicate with each other. But it's very hard to understand why we are doing this motion when we are doing this motion. Right. I don't know why I'm sitting like this. I'm not trying to send you... I mean, I guess I'm sending you a signal, but I don't know what it means. Exactly. Usually we are using our gestures, but it's very hard to define why we are using it. We are not doing this consciously, right? This is nonverbal gestures. I mean, I'm just like trying to sit in a way that looks good on camera. I like your socks, by the way. Thank you. I'm trying to show off my socks. That's why I'm sitting like this. And that's the really weird thing about body language. It's something we all do unconsciously and understand innately, but if I were to stop and try to tell you why I'm holding my shoulders a certain way or moving my hands, I wouldn't understand it to be able to explain it. So if we can't explain it to each other, how can we possibly teach it to machines? Well, it turns out we can teach them the same way we've been teaching a lot of other artificial intelligence systems. Machine learning, specifically a technique called deep learning, has produced a number of huge breakthroughs in the last few years. A very simplistic view of it is you feed machines a lot of data about a certain subject, like say driving a car. Feed it enough data about how we drive and it will be able to start understanding the system at play and react in a similar fashion to the way we do. Most recently, Google's DeepMind watched a few years worth of TV shows to understand how to read human lips. Now it can do that better than most experts. The Panoptix Studio hopes that given enough visual data showing how humans interact, its deep learning system will be able to tease out the structure of that elaborate code, making it possible for computers to read and write our body language, possibly understanding a facet of our communication better than we do ourselves. So if all this sounds familiar, you have probably been watching the new HBO series Westworld. It's about a park filled with robots that can so vividly replicate human behavior it becomes impossible to distinguish who's a real human and who's not. So Westworld is science fiction, sure, but do a quick YouTube search for lifelike robots and you'll find we've actually made some incredible progress in the last few decades. And some stuff to fuel your nightmares. By the time you're grown up, I'll be as smart as a real person and we will be like brothers. I do not know if you will remember me then, but I'll never forget you. Someday I'll come and find you and we'll be good friends. But the benefits of these social robots could be immense. Look at Milo. He's a robot that's being used to help kids with autism. He can help these children identify social cues and emotional reactions that they can sometimes struggle to express and interpret themselves. And robots will also help us care for the elderly, so less Westworld and more Franken the Robot. Beyond helping accomplish physical tasks, future robots could become vital emotional supporters as well. Imagine instead of a roboceptionist, Tank could help someone suffering from Alzheimer's socialize on a more regular basis, identifying their mood and reacting appropriately. There's many components that need to happen before this works. I mean AI in particular needs to get better, right? But I think that there will be a big advantage in terms of how we will treat machines once they start respecting at least, for example, whether we are paying attention to them or whether we are open to an interaction and other sort of social signals that we use in our daily conversations. But if all this talk of lifelike robots makes you nervous about the coming of some sky net that'll wipe out humanity, consider this. Teaching robots this social behavior might be the best way for us to foster a better understanding between humans and their creations. So one way of putting it is that we use machines and even robots as tools currently, but what we will achieve when they interact with us as almost equals is we will find collaborators. Also we won't start using machines as tools but rather as agents that we can collaborate with. So I think in those terms that that will be very important. So if we want to integrate machines and it's eventually going to happen, they should at least respond to us in a way that we appreciate. This research is already being used by a company you probably know well, Facebook. The social networking giant and its Oculus division are working with researchers from the Panoptix studio to find ways to make avatars in VR more accurate and social. This could make communication with a friend more intimate or it could give Siri a body that can walk and talk just like a real person.
 If you ever find yourself in Pittsburgh and want to catch a glimpse of the future,
 Carnegie Mellon University is probably a good place to start.
 To get inside the robotics lab, you'll need to walk past Tank. He's a robo-ceptionist and,
 well, actually Tank is really important to this story, but he isn't what I want
 to show you right now. This is.
 Looking at this large geodesic dome covered in wires, you can't help but feel
 like you're staring into a time machine, and in a way you kind of are. But rather
 than traveling through time, this panoptic studio is capturing it. Using an
 array of cameras, researchers here are able to capture and then replicate any
 action into a 3D model. From a dance to a cello performance, it can perfectly
 reconstruct these moments in time. And while that's pretty impressive by itself,
 it's just the starting point in a quest to decipher an age-old code.
 So basically you can see that this is a kind of geodesic dome. Looks like a sphere.
 We specifically designed this architecture so that we can put cameras
 as uniform as possible. And you can see also that we specially modularized the
 system so that we only designed one panel and we can duplicate the panels.
 One panel is composed of 24 small cameras and one high-resolution cameras
 and they are saved in the local hard drives and in the end all the local data
 are they are transmitted to the NAS. And so that sounds like a ton of data, like
 how much data is in one minute of capture? So one minute of data is about
 600 gigabytes. That's huge data. That's a big file. In fact, to this point, the
 Panoptix Studio has captured over one petabyte of data. That's about a million
 gigabytes. And the latest is, well, me. Ready? Can you see me? Yeah? So while I'm
 dancing, those 500 cameras are capturing every movement I make. And after a couple
 of weeks of processing, it looks something like this. This point cloud
 view is not only showing a bunch of tracking points, it's tracing the paths
 they could take while I'm moving around. All right, glad I stretched before that.
 Now, motion capture by itself isn't anything new. Similar technology has been
 used to bring digital characters to life in films for years now. Motion capture
 technology has gotten so good that an actor like Andy Serkis can really inject
 life into a fully digital character. But to do so requires a full motion capture
 suit with tracking markers placed all over your body, even painted on your face.
 Now, this works well enough for trying to bring a dragon to life, but it's hard to
 feel natural in that kind of equipment. The Panoptix Studio, in contrast, allows
 for similar tracking, but without any of the physical markers. You can just
 freely move your body in a way that feels natural and unrestricted, which is
 a necessity for them to capture what they are really looking for. A really good
 quote from Edward Sapir, who is a linguist. He says, there's an elaborate
 and secret code that is known to no one, but understood by all. And he's referring
 to all the gestures that we use to communicate. And that's kind of the code
 that we're trying to sort of crack or understand. Our goal, essentially, is to
 endow machines with the capacity to understand these social interactions.
 So that's where Tank comes back into play. You remember Tank, the roboceptionist?
 He can see me via camera and react to my presence, but he can't understand more
 than what I type to him. So when I start to get frustrated that he isn't helping
 me find the Panoptix Studio, he can't read that frustration or react in a way
 that helps ease my tension, which kind of makes him bad at his job. Machines have a
 very poor understanding of what all those things are. So, for example, if I'm
 not paying attention to you or if I'm looking at you and paying attention,
 those are the kinds of things that machines just don't respond to right now.
 And that's kind of a big deal. Studies have suggested that when we communicate
 with each other, more than half of the message we're sending is based on the
 way we move. Our posture, facial expressions, and gestures can at times
 convey more than our words alone. Here, let me show you. So now you can see me
 and you can read what my body is telling you, even if my words are saying
 something different. Maybe from my posture you can tell I'm in a cramped
 little sound booth, or you can tell I'm excited about this story, that I'm
 nervous about reading from a script, that I'm hopelessly improvising into a camera
 to try and give you a sense of how expressive body language can be. When we
 communicate with each other, we use many interesting hand gestures, which is really
 important to communicate with each other. But it's very hard to understand why we
 are doing this motion when we are doing this motion. Right. I don't know why I'm
 sitting like this. I'm not trying to send you... I mean, I guess I'm sending you a signal, but I don't know what it means.
 Exactly. Usually we are using our gestures, but it's very hard to define why we are
 using it. We are not doing this consciously, right? This is nonverbal gestures.
 I mean, I'm just like trying to sit in a way that looks good on camera. I like your socks, by the way.
 Thank you. I'm trying to show off my socks. That's why I'm sitting like this.
 And that's the really weird thing about body language. It's something we all do
 unconsciously and understand innately, but if I were to stop and try to tell
 you why I'm holding my shoulders a certain way or moving my hands, I
 wouldn't understand it to be able to explain it. So if we can't explain it to
 each other, how can we possibly teach it to machines? Well, it turns out we can
 teach them the same way we've been teaching a lot of other artificial
 intelligence systems. Machine learning, specifically a technique called deep
 learning, has produced a number of huge breakthroughs in the last few years. A
 very simplistic view of it is you feed machines a lot of data about a certain
 subject, like say driving a car. Feed it enough data about how we drive and it
 will be able to start understanding the system at play and react in a similar
 fashion to the way we do. Most recently, Google's DeepMind watched a few years
 worth of TV shows to understand how to read human lips. Now it can do that
 better than most experts. The Panoptix Studio hopes that given enough visual
 data showing how humans interact, its deep learning system will be able to
 tease out the structure of that elaborate code, making it possible for
 computers to read and write our body language, possibly understanding a facet
 of our communication better than we do ourselves.
 So if all this sounds familiar, you have probably been watching the new HBO
 series Westworld. It's about a park filled with robots that can so vividly
 replicate human behavior it becomes impossible to distinguish who's a real
 human and who's not. So Westworld is science fiction, sure, but do a quick
 YouTube search for lifelike robots and you'll find we've actually made some
 incredible progress in the last few decades. And some stuff to fuel your
 nightmares. By the time you're grown up, I'll be as smart as a real person and we will be like brothers. I do not know if you will
 remember me then, but I'll never forget you. Someday I'll come and find you and we'll be good friends.
 But the benefits of these social robots could be immense. Look at Milo. He's a
 robot that's being used to help kids with autism. He can help these children
 identify social cues and emotional reactions that they can sometimes
 struggle to express and interpret themselves. And robots will also help us
 care for the elderly, so less Westworld and more Franken the Robot. Beyond helping
 accomplish physical tasks, future robots could become vital emotional supporters
 as well. Imagine instead of a roboceptionist, Tank could help someone
 suffering from Alzheimer's socialize on a more regular basis, identifying their
 mood and reacting appropriately. There's many components that need to happen
 before this works. I mean AI in particular needs to get better, right? But I think
 that there will be a big advantage in terms of how we will treat machines once
 they start respecting at least, for example, whether we are paying attention
 to them or whether we are open to an interaction and other sort of social
 signals that we use in our daily conversations.
 But if all this talk of lifelike robots makes you nervous about the coming of
 some sky net that'll wipe out humanity, consider this. Teaching robots this
 social behavior might be the best way for us to foster a better understanding
 between humans and their creations. So one way of putting it is that we use
 machines and even robots as tools currently, but what we will achieve when
 they interact with us as almost equals is we will find collaborators. Also we
 won't start using machines as tools but rather as agents that we can collaborate
 with. So I think in those terms that that will be very important. So if we want to
 integrate machines and it's eventually going to happen, they should at least
 respond to us in a way that we appreciate. This research is already being
 used by a company you probably know well, Facebook. The social networking giant and
 its Oculus division are working with researchers from the Panoptix studio to
 find ways to make avatars in VR more accurate and social. This could make
 communication with a friend more intimate or it could give Siri a body
 that can walk and talk just like a real person.