 When I was a kid, I was obsessed with visual effects movie specials. How they made the Rancor monster or how that water tentacle in the abyss worked, that's the stuff that I wanted to know about. And most of the time, the company behind it was Industrial Light & Magic. Now that same company, they're getting into virtual reality, they're getting into augmented reality and interactive experiences. And the group is called ILMxLAB. The goal is to create a sort of digital Disneyland, an interconnected world of experiences that let people immerse themselves in their favorite stories to whatever degree they want. This lets the mocap system sort of see where you are in space and make this view custom to your location. So the character that you're seeing. Holy shit. Ha, dump dump dump. I'm sorry, I can't hang out with the dinosaur. What were you saying? ILMxLAB came out of a collaboration between three companies, really. It's Lucasfilm Story Group and ILM for the visuals and Skywalker Sound for the sound. And all of us were exploring, on top of the advanced development group's technology, the area of VR and AR and immersive technologies like these. So basically right now what I'm seeing is a dinosaur right here at my hand, basically resting its chin on my hand right now in a three-dimensional space. It's pretty wild. It's going to bite my head off. So what you're seeing actually is Chris over there performing as the raptor. And so he's responding to you. And his performance gets translated over to the raptor's motion. And all in real time, so this is nuts. Yeah, all in real time. Because it's ILM, they've got 40 years of visual effects experience to work with. And because it's Lucasfilm, they've got some pretty big movies to play with. You know the ones. We have different experiments that we're running right now to see which ones are going to be the most successful. But the amazing opportunity here is to work with the filmmakers and the production art department and all of the creators of both the standalone films, the anthology films, and the episode 7, 8, 9 films. All those creators actually work right here in the building and spend time here working hand in hand with us. So we have the opportunity to show them the latest technology, show them the latest experiments in terms of storytelling, and riff off of each other. Try the next generation of things to see what is successful and what isn't and what do people want to experience. See Theopro's shorter than I thought he was going to be. Just what everyone says. This is another kind of toy related to some of our experiments with doing set design stuff. You can go and do virtual building block things, just rough layout. I have to imagine the implications here for funneling back through to ILM. If a filmmaker wants to tour a virtual set before they shoot in it or even change things, have the DP come in. That's going to be an incredible opportunity. That's actually one of the main real world applications we use for this. One of the tools we created for Gareth on his film was to help him pre-visualize some of his sets. He's talking about Gareth Edwards, who's directing Star Wars Rogue One, and he was looking at one of the sets through a virtual reality headset like the Oculus Rift. He was in virtual reality for the first time and this was a very rough set. It wasn't to a high degree of finish or anything. He was in there exploring and a couple minutes in he said, this looks way better in real life. I leaned over to Doug Chang, who's a production designer, and I said, Doug, have you built this yet? Is he talking about real life or is he talking about... He's like, no, he's talking about the VR headset. When he, two minutes into having the headset on, referred to it as real life, I was like, okay, we're on to something here. It's fun, yeah. Now, can we have the Velociraptor hang out with 3PO and R2? Take about five minutes to do that over here. My whole career has been about creative development, it's been about designing moments and perspectives upon moments and things like this. By things like this, he means things like this. If you saw Speed Racer, you saw John's work pioneering a hyper-stylized visual aesthetic dubbed photo anime. And then this scene from The Matrix Reloaded used a technique called volumetric cinematography, also pioneered by John. And oh yeah, bullet time, that was John too. The actual experience of The Matrix itself started expanding my thought on the very topic that the movie was about. I joined that effort with the Wachowskis partly because I thought it was so well written, how they considered virtual reality completely sublime and in the perfect state. It's flawless, it's a flawless simulation. Our approach is, as each film went on and then the years after, was to really try to make it in the same sort of way that it might actually be made someday. To try to find some techniques that were about capturing reality and then of course it would end up as cinema, composed and all that. But yeah, we went deep into a lot of exploration and a lot of things fell out of that that are super relevant to me today and what we're doing here. And a lot of those friends, younger people then, have found their way into some pretty interesting places throughout the community that we are watching right now. For a while he left film to start his own company and focus strictly on AR and VR. That's him there, covered in augmented reality tank goop. And if it seems like John is trying to build The Matrix for real, that's because John is kind of trying to build The Matrix for real. To quote his LinkedIn page, I'm trying to design and enable platforms that lead to The Matrix, minus machine dominance. So that's reassuring. And he's the kind of guy that really might pull it off. Just listen to him talk about the future of filmmaking in this behind the scenes featurette from 1999. It's all baby steps towards something much larger that won't be really commonplace for a few years. We're talking about cameras that are now broken from the subject matter that are virtual. Fifteen years later. So this is a virtual camera. It's basically a window into the world. So as a director you can see what's going on in your peripheral vision. As you're framing up on something you can sort of pan over to action that's happening off camera without having to rehearse it because you just see it. X-Labs is also making stuff that you might actually get to play with soon. So all of this is actually generated in real time from the cloud and it streams to this iPad in my hand and that lets us deliver a level of fidelity that's way beyond what this tablet could do, approaches movies or TV. But at the touch of a button we can let the user take the cinematic borders away and everything is fully interactive. We've been experimenting a lot with story, experimenting a lot with how we get people to relate, what is their persona in this environment. That is our focus. If we can solve those creative problems or at least make progress against those challenges then I think the demand and the audience will find a way to experience our stories. And at the moment time is frozen so if we use this slide over here we can start to add time back and now we can actually watch the whole section unfold while still having full control of the camera. As well as camera control we let you interact with the environment and characters so if you tap a stormtrooper we'll go into an orbit around this character and we can actually follow him through the whole piece and see everything that he sees. Quite literally we can jump into his perspective and now everything would play out with us here. We have this prototype device in the back that's a depth sensor. It's kind of like Kinect. It's actually the same type of sensor. Because we have depth we can actually let you walk into the scene, move around and be your own cameraman. Until you run into a wall. We do think that people want to go further, right? They want to get deeper. So we revisit a gate in the movie and observe our story sculpture. It's like a work of art. It's like a sculpture. This next question is for film nerds only. So I'm seeing here down here it says it's running about 24 frames a second. Why that versus 30 or 60? So as you know cinema is 24 frames a second. In fact we actually physically model the properties of a camera so when you run it at 24 you get just the right amount of motion blur that gives you that cinematic feel. And when we had that we did a lot of A B testing with people internally like which one do you prefer and overwhelmingly everybody thought the 24 was more cinematic. The whole experience is just under two minutes. You can re-experience it an infinite number of times in infinite number of ways. Anything that you may see should be things you're allowed to experience. Not necessarily, you know, I mean I'm sure there are versions of this that involve gaming and competition but really what I'm talking about is kind of being in the shoes of a character like a character you saw in the story. For instance I could be watching the speeder bike scene in Return of the Jedi, hit a button and then suddenly. Oh whoa, I'm like on a speeder bike. There's buttons here and there's weird dials here. I put it in your hand and so that's your speed control. Whoa! And then lean sideways, yeah. You can pick it up completely natural. It's like a motorbike. So is it me leaning or is it the phone? Whoa! If you lean forward you'll be fine. Leaning backward will gain altitude. This is fun. This was very much like a Skunk Works project. Like we were building this other experience and in the hours after work kind of I decided let's just hook this up to the Oculus and see what works. And like the cinematic cuts for example that you've seen we still need to figure out what the real cinematic language needs to be. Like when should you cut? Should you ever cut? There's a lot of like new hundred years of cinema learning that needs to be relearned in this medium. He brings up an interesting point. It took us a really long time to develop the ideas of what would make cinema work. Early on people thought you had to show scenes in their entirety before they realized you could compress time with cuts and evoke emotion through the juxtaposition of different imagery. This is called montage. And also create moods with lighting and framing and a million other things that we don't even think about anymore. Like you inherently know if I use a transition like this. This is kind of a cheesy low budget video. Our understanding of film vocabulary is so advanced now you can watch a heavily edited scene like this with no problem. About 120 years ago people were just excited to see moving pictures. In fact according to legend people saw this train headed straight towards them and jumped right out of their seats. Sound familiar? Human beings in general are comfortable with passive media storytelling. Storytelling in general right. It's been with us forever. And if you want to draw the masses in you have to do it in the most, you know you have to baby step them deeper and deeper into what will become some pretty sophisticated types of engagement. Alright yeah this concludes the VR portion. At the heart of it of course you know is the story group and I'm lucky in so far as I work with them directly. The story group is kind of like Lucasfilm's creative brain trust. It's the team responsible for shaping and shepherding every piece of Star Wars content out there so they're all part of the same big story no matter what the medium. That goes for new movies, TV, video games, the VR experiences and any future form of entertainment not yet discovered. Real transmedia stuff that we've never been able to do before because in the past we've had one camp that was making the films and another camp that was doing the publishing and the licensing and the published fiction. So now by putting them together we're able to be a lot more coherent, a lot more cohesive, a lot more connected in our stories going forward. That's the kind of future X-Lab is envisioning. One with interconnected experiences that go from films to VR to your tablet and back. Where movies are shot on virtual locations that you can then go visit with your friends and hang out in. And because it's all happening under one roof it's part of the same giant interactive Star Wars story. We'll see how far we can get but I think that's the new model. Deep media, highly connected, smashed silo structures and all tapped and hubbed off of the origin creators. You need to be close to that to understand how you can get emotion. So it's the X factor around here. If AR ends up being huge and VR ends up being medium sized that works out just fine for us. We've got lots of plays in both. Even if we were to only leverage this technology for the filmmakers to be immersed in their environments that's really useful too. So they could just stay behind the scenes and that would work out fine. But the team at X-Lab clearly have bigger plans. And they're taking their inspiration from somewhere that should seem obvious. We really, really like working with the Imagineers because Walt Disney is the godfather of experience UX and all. The parks in a way are the first prototype for immersive entertainment. And so there's some real foundational stuff there and their method. This is exactly the same program modernized. This is like a 2.0 supercharged version of the same philosophy. Once the destination is built you can wander around in it, you can be social in it, you can do other things, you can play in it. The metaverse will begin like one room at a time. And what you put in the room and what people do in the room can be the beginning of many things. It's just like in the crafting of what this room is going to be about. So I'm hoping and assuming that people are going to create brilliant rooms where there will be interactions and it could be like just hanging out with my friends. But folks will be more creative than that. They'll create mechanics that make people interact in new ways. I'm not sure if you've tried any of the social VR stuff but it's pretty unique to be all in a common virtual space together and you're all over the world, right? But you are together. This is one of those virtual spaces he's talking about. That's John. It feels very interesting. You quickly forget that you're talking to someone who is on the other side of the planet. You're just like there. A lot of kids already really pretty much live a life adorned as an avatar already, right? So let's show up. Let them go nuts.
 When I was a kid, I was obsessed with visual effects movie specials.
 How they made the Rancor monster or how that water tentacle in the abyss worked, that's
 the stuff that I wanted to know about.
 And most of the time, the company behind it was Industrial Light & Magic.
 Now that same company, they're getting into virtual reality, they're getting into augmented
 reality and interactive experiences.
 And the group is called ILMxLAB.
 The goal is to create a sort of digital Disneyland, an interconnected world of experiences that
 let people immerse themselves in their favorite stories to whatever degree they want.
 This lets the mocap system sort of see where you are in space and make this view custom
 to your location.
 So the character that you're seeing.
 Holy shit.
 Ha, dump dump dump.
 I'm sorry, I can't hang out with the dinosaur.
 What were you saying?
 ILMxLAB came out of a collaboration between three companies, really.
 It's Lucasfilm Story Group and ILM for the visuals and Skywalker Sound for the sound.
 And all of us were exploring, on top of the advanced development group's technology, the
 area of VR and AR and immersive technologies like these.
 So basically right now what I'm seeing is a dinosaur right here at my hand, basically
 resting its chin on my hand right now in a three-dimensional space.
 It's pretty wild.
 It's going to bite my head off.
 So what you're seeing actually is Chris over there performing as the raptor.
 And so he's responding to you.
 And his performance gets translated over to the raptor's motion.
 And all in real time, so this is nuts.
 Yeah, all in real time.
 Because it's ILM, they've got 40 years of visual effects experience to work with.
 And because it's Lucasfilm, they've got some pretty big movies to play with.
 You know the ones.
 We have different experiments that we're running right now to see which ones are going to be
 the most successful.
 But the amazing opportunity here is to work with the filmmakers and the production art
 department and all of the creators of both the standalone films, the anthology films,
 and the episode 7, 8, 9 films.
 All those creators actually work right here in the building and spend time here working
 hand in hand with us.
 So we have the opportunity to show them the latest technology, show them the latest experiments
 in terms of storytelling, and riff off of each other.
 Try the next generation of things to see what is successful and what isn't and what
 do people want to experience.
 See Theopro's shorter than I thought he was going to be.
 Just what everyone says.
 This is another kind of toy related to some of our experiments with doing set design stuff.
 You can go and do virtual building block things, just rough layout.
 I have to imagine the implications here for funneling back through to ILM.
 If a filmmaker wants to tour a virtual set before they shoot in it or even change things,
 have the DP come in.
 That's going to be an incredible opportunity.
 That's actually one of the main real world applications we use for this.
 One of the tools we created for Gareth on his film was to help him pre-visualize some
 of his sets.
 He's talking about Gareth Edwards, who's directing Star Wars Rogue One, and he was looking at
 one of the sets through a virtual reality headset like the Oculus Rift.
 He was in virtual reality for the first time and this was a very rough set.
 It wasn't to a high degree of finish or anything.
 He was in there exploring and a couple minutes in he said, this looks way better in real
 life.
 I leaned over to Doug Chang, who's a production designer, and I said, Doug, have you built
 this yet?
 Is he talking about real life or is he talking about... He's like, no, he's talking about
 the VR headset.
 When he, two minutes into having the headset on, referred to it as real life, I was like,
 okay, we're on to something here.
 It's fun, yeah.
 Now, can we have the Velociraptor hang out with 3PO and R2?
 Take about five minutes to do that over here.
 My whole career has been about creative development, it's been about designing moments and perspectives
 upon moments and things like this.
 By things like this, he means things like this.
 If you saw Speed Racer, you saw John's work pioneering a hyper-stylized visual aesthetic
 dubbed photo anime.
 And then this scene from The Matrix Reloaded used a technique called volumetric cinematography,
 also pioneered by John.
 And oh yeah, bullet time, that was John too.
 The actual experience of The Matrix itself started expanding my thought on the very topic
 that the movie was about.
 I joined that effort with the Wachowskis partly because I thought it was so well written,
 how they considered virtual reality completely sublime and in the perfect state.
 It's flawless, it's a flawless simulation.
 Our approach is, as each film went on and then the years after, was to really try to
 make it in the same sort of way that it might actually be made someday.
 To try to find some techniques that were about capturing reality and then of course it would
 end up as cinema, composed and all that.
 But yeah, we went deep into a lot of exploration and a lot of things fell out of that that
 are super relevant to me today and what we're doing here.
 And a lot of those friends, younger people then, have found their way into some pretty
 interesting places throughout the community that we are watching right now.
 For a while he left film to start his own company and focus strictly on AR and VR.
 That's him there, covered in augmented reality tank goop.
 And if it seems like John is trying to build The Matrix for real, that's because John is
 kind of trying to build The Matrix for real.
 To quote his LinkedIn page, I'm trying to design and enable platforms that lead to The
 Matrix, minus machine dominance.
 So that's reassuring.
 And he's the kind of guy that really might pull it off.
 Just listen to him talk about the future of filmmaking in this behind the scenes featurette
 from 1999.
 It's all baby steps towards something much larger that won't be really commonplace for
 a few years.
 We're talking about cameras that are now broken from the subject matter that are virtual.
 Fifteen years later.
 So this is a virtual camera.
 It's basically a window into the world.
 So as a director you can see what's going on in your peripheral vision.
 As you're framing up on something you can sort of pan over to action that's happening
 off camera without having to rehearse it because you just see it.
 X-Labs is also making stuff that you might actually get to play with soon.
 So all of this is actually generated in real time from the cloud and it streams to this
 iPad in my hand and that lets us deliver a level of fidelity that's way beyond what this
 tablet could do, approaches movies or TV.
 But at the touch of a button we can let the user take the cinematic borders away and everything
 is fully interactive.
 We've been experimenting a lot with story, experimenting a lot with how we get people
 to relate, what is their persona in this environment.
 That is our focus.
 If we can solve those creative problems or at least make progress against those challenges
 then I think the demand and the audience will find a way to experience our stories.
 And at the moment time is frozen so if we use this slide over here we can start to add
 time back and now we can actually watch the whole section unfold while still having full
 control of the camera.
 As well as camera control we let you interact with the environment and characters so if
 you tap a stormtrooper we'll go into an orbit around this character and we can actually
 follow him through the whole piece and see everything that he sees.
 Quite literally we can jump into his perspective and now everything would play out with us
 here.
 We have this prototype device in the back that's a depth sensor.
 It's kind of like Kinect.
 It's actually the same type of sensor.
 Because we have depth we can actually let you walk into the scene, move around and be
 your own cameraman.
 Until you run into a wall.
 We do think that people want to go further, right?
 They want to get deeper.
 So we revisit a gate in the movie and observe our story sculpture.
 It's like a work of art.
 It's like a sculpture.
 This next question is for film nerds only.
 So I'm seeing here down here it says it's running about 24 frames a second.
 Why that versus 30 or 60?
 So as you know cinema is 24 frames a second.
 In fact we actually physically model the properties of a camera so when you run it at 24 you get
 just the right amount of motion blur that gives you that cinematic feel.
 And when we had that we did a lot of A B testing with people internally like which one do you
 prefer and overwhelmingly everybody thought the 24 was more cinematic.
 The whole experience is just under two minutes.
 You can re-experience it an infinite number of times in infinite number of ways.
 Anything that you may see should be things you're allowed to experience.
 Not necessarily, you know, I mean I'm sure there are versions of this that involve gaming
 and competition but really what I'm talking about is kind of being in the shoes of a character
 like a character you saw in the story.
 For instance I could be watching the speeder bike scene in Return of the Jedi, hit a button
 and then suddenly.
 Oh whoa, I'm like on a speeder bike.
 There's buttons here and there's weird dials here.
 I put it in your hand and so that's your speed control.
 Whoa!
 And then lean sideways, yeah.
 You can pick it up completely natural.
 It's like a motorbike.
 So is it me leaning or is it the phone?
 Whoa!
 If you lean forward you'll be fine.
 Leaning backward will gain altitude.
 This is fun.
 This was very much like a Skunk Works project.
 Like we were building this other experience and in the hours after work kind of I decided
 let's just hook this up to the Oculus and see what works.
 And like the cinematic cuts for example that you've seen we still need to figure out what
 the real cinematic language needs to be.
 Like when should you cut?
 Should you ever cut?
 There's a lot of like new hundred years of cinema learning that needs to be relearned
 in this medium.
 He brings up an interesting point.
 It took us a really long time to develop the ideas of what would make cinema work.
 Early on people thought you had to show scenes in their entirety before they realized you
 could compress time with cuts and evoke emotion through the juxtaposition of different imagery.
 This is called montage.
 And also create moods with lighting and framing and a million other things that we don't even
 think about anymore.
 Like you inherently know if I use a transition like this.
 This is kind of a cheesy low budget video.
 Our understanding of film vocabulary is so advanced now you can watch a heavily edited
 scene like this with no problem.
 About 120 years ago people were just excited to see moving pictures.
 In fact according to legend people saw this train headed straight towards them and jumped
 right out of their seats.
 Sound familiar?
 Human beings in general are comfortable with passive media storytelling.
 Storytelling in general right.
 It's been with us forever.
 And if you want to draw the masses in you have to do it in the most, you know you have
 to baby step them deeper and deeper into what will become some pretty sophisticated types
 of engagement.
 Alright yeah this concludes the VR portion.
 At the heart of it of course you know is the story group and I'm lucky in so far as I work
 with them directly.
 The story group is kind of like Lucasfilm's creative brain trust.
 It's the team responsible for shaping and shepherding every piece of Star Wars content
 out there so they're all part of the same big story no matter what the medium.
 That goes for new movies, TV, video games, the VR experiences and any future form of
 entertainment not yet discovered.
 Real transmedia stuff that we've never been able to do before because in the past we've
 had one camp that was making the films and another camp that was doing the publishing
 and the licensing and the published fiction.
 So now by putting them together we're able to be a lot more coherent, a lot more cohesive,
 a lot more connected in our stories going forward.
 That's the kind of future X-Lab is envisioning.
 One with interconnected experiences that go from films to VR to your tablet and back.
 Where movies are shot on virtual locations that you can then go visit with your friends
 and hang out in.
 And because it's all happening under one roof it's part of the same giant interactive Star
 Wars story.
 We'll see how far we can get but I think that's the new model.
 Deep media, highly connected, smashed silo structures and all tapped and hubbed off of
 the origin creators.
 You need to be close to that to understand how you can get emotion.
 So it's the X factor around here.
 If AR ends up being huge and VR ends up being medium sized that works out just fine for
 us.
 We've got lots of plays in both.
 Even if we were to only leverage this technology for the filmmakers to be immersed in their
 environments that's really useful too.
 So they could just stay behind the scenes and that would work out fine.
 But the team at X-Lab clearly have bigger plans.
 And they're taking their inspiration from somewhere that should seem obvious.
 We really, really like working with the Imagineers because Walt Disney is the godfather of experience
 UX and all.
 The parks in a way are the first prototype for immersive entertainment.
 And so there's some real foundational stuff there and their method.
 This is exactly the same program modernized.
 This is like a 2.0 supercharged version of the same philosophy.
 Once the destination is built you can wander around in it, you can be social in it, you
 can do other things, you can play in it.
 The metaverse will begin like one room at a time.
 And what you put in the room and what people do in the room can be the beginning of many
 things.
 It's just like in the crafting of what this room is going to be about.
 So I'm hoping and assuming that people are going to create brilliant rooms where there
 will be interactions and it could be like just hanging out with my friends.
 But folks will be more creative than that.
 They'll create mechanics that make people interact in new ways.
 I'm not sure if you've tried any of the social VR stuff but it's pretty unique to be all
 in a common virtual space together and you're all over the world, right?
 But you are together.
 This is one of those virtual spaces he's talking about.
 That's John.
 It feels very interesting.
 You quickly forget that you're talking to someone who is on the other side of the planet.
 You're just like there.
 A lot of kids already really pretty much live a life adorned as an avatar already, right?
 So let's show up.
 Let them go nuts.