 Could light-field displays be the tech that replaces our smartphones? In recent years, mixed reality headsets and so-called smart glasses have been promising to give us an interactive experience right in front of our eyes, instead of having to stare down at our phones. But the glasses and headsets that have come out have either been awkward or have found a niche firmly in the enterprise market, which is what a company called Avagant thinks it can change. This is a mixed reality headset. It's called mixed reality because you can still see the physical world around you, but you're also seeing digital objects laid over the real world. And this is Avagant's newest headset prototype, the one that the company has built to show hardware partners what it can do, specifically with light-field displays. Now like Microsoft HoloLens, the Avagant headset shows digital objects that appear to be floating in space. Unlike HoloLens, this headset prototype is tethered to a PC, running Unity. This is the optics engine that we have. This is what, you know, creates the entire image. This has our retinal imaging technology, the same technology we've used previously in the Glyph. So we're using micro-mirror technology to project images to your retina, and also inside of this we've incorporated our new light-field optics in here. We're simultaneously sending you multiple focal planes. What that means is we're not actually tracking your eyes, which means you can even do this with a single eye. So if you close one eye, you can feel the focus change with a single eye between the two points. So what can you do with something like the Avagant headset? Well, I navigated the solar system and saw the room around me transformed into a giant fish tank. The experience was partly enabled by camera tracking systems, placed either in the corners of the room or directly on the headset. I also tried a couple of apps that have been shown in promotional videos but haven't been experienced by many people in real life before. One app was a beating human heart, and it was originally made for HoloLens, and another app let me hold digital objects right in the palm of my hand. Alligator. Rhino. Oh, a syringe. Oh, that's creepy. Ew, it's going into my hand. Oh, I have a syringe stuck in my hand. This is bizarre. Okay, so in order to get a sense of what Avagant is trying to do, it helps to understand how light field technology works, and it comes down to how we perceive depth. If you hold something up in front of your face, for example, what you're seeing is the light that bounces off its surface and passes into your eyes from multiple angles. Your brain combines the 2D image from each eye and uses visual cues like the rotation and focus of your eyes to create a sense of depth. You'll also notice that you can really easily shift your focus from the thing that's right in front of your face to the stuff that's in the background. When you start looking at digital images through smart glasses or mixed reality headsets, you're usually seeing a flat 2D image with a fixed focal point. And when you've got a display mashed right up against your face, it just doesn't feel natural. By using light field optics in its headset, or what Avagant calls its multifocal plane approach, Avagant says it's essentially cutting up the images in front of you into different focal planes and sending that light into your eyes. So when you go to look at different digital objects, it's supposed to replicate the way your eyes would naturally focus without a headset on. So right now as I'm looking at you, you're in focus, but the arcade game behind you is not. But if I were to keep moving my eyes back and forth, they're sort of naturally adjusting. And you're saying that when someone wears an Avagant headset, that your technology lets that happen sort of naturally. It doesn't feel like you're straining your eyes. That's right. So when you try the light field technology, it should not only feel more realistic, it also should feel a lot more comfortable and natural for your eyes. There are some elements of Avagant's technology that the company declined to talk about, like the number of planes it uses in this multifocal plane approach, and exactly how it's creating this continuous focal experience. One industry expert pointed out to me that terms like light fields and holograms can be a little overused. Generally speaking, the best way to look at these kinds of solutions is to consider them a practical approximation of an analog light field display. Right now, Avagant seems confident that its technology is superior to at least some of its competitors out there. Although it's hard to say how it stacks up to other companies that are holding their tech pretty close to the vest. So how is that different from what, something like a HoloLens? They're a fixed focus, right? So the image that you see is fixed at a certain distance. Now they try to use stereoscopic 3D effects to make your eyes converge at different angles to give you a sense of 3D, but they're not really changing the focus of that image. That's okay when things are generally farther away, but that whole display and experience breaks down the second that you get within about a meter. So you think about, if I want to hold an object in my hands, if I want to have a face-to-face conversation with a virtual person, you actually really can't do any of those things unless you have a light field display, unless you solve those focus problems. How does it compare to what Magic Leap is doing? We don't know exactly what Magic Leap is doing. Have you seen Magic Leap? No, I haven't. Maybe you guys have a better idea. I'm sure you've read a lot about it. Certainly. I mean, we can just see what's in the news. We don't know exactly what they're doing, but at least from what's public, I would say that we both believe that light field is a critical part of the experience. You might have heard of Magic Leap by now. It's a super secretive company in Florida that has raised more than a billion dollars to date to finance its digital light field technology. I reached out to Magic Leap and not surprisingly, the company didn't want to share specific details on their technology. But one thing that Magic Leap has said is that they're building what you might call a full stack solution. They're building the computing system, the headset, the software. They're even working with content creators. Avagon's plan for now isn't to build a product it's going to ship directly to consumers, but to partner with other hardware companies who will use Avagon's optics in their own headsets. This could come as soon as 2018, Avagon says, but it could take longer to get to the consumer market. But when that happens, they believe there's a very real chance that something like this could replace the smartphone as we know it. We always knew that as things were moving to mobile devices like phones and tablets, that the next evolution with that was going to this idea of the ambient computer. Something that I can put on like glasses and wear and suddenly I'm surrounded by these experiences and not limited to the screens that we can fit in our pockets or fit in our bags. So do you really envision a world in which people are just walking around wearing air glasses all the time, even if they're not currently people who wear glasses or something on their face? I actually do. I think the smartphone as we know it today will get replaced by smart glasses. And once the industry overcomes some of the challenges that still exist today to make the devices much lighter and less bulky, I think yes, people will be wearing those glasses for an extended period of time. This is the thing that a lot of technologists are talking about right now. What's the next big thing beyond smartphones? Is there a thing beyond mobile or will we all be staring at the glass rectangles in our hands for many more years to come? If we really are approaching a post-mobile world, then Avagon is trying to position itself at the forefront of this. And while a lot of headsets so far haven't really caught on with the mass market, Avagon thinks there's a time in the very near future when these things are just like eyeglasses, that you just carry them in your pocket and slip them on instead of staring at your smartphone. It's a bold idea of a computing future, and it's going to take a lot more than optics to make it work, things like apps and content that make people want to wear displays on their faces. It might be a future that we're not quite ready for, but it might also be one that's happening anyway. You must never get tired of looking at people while they wear this thing. You get very used to it. You know, it's the typical look, you sit there, you're very quiet, jaws always open a little bit, just like you saw for the first time. What's funny is I actually can't hear you at all right now. I just see this.
 Could light-field displays be the tech that replaces our smartphones?
 In recent years, mixed reality headsets and so-called smart glasses have been promising
 to give us an interactive experience right in front of our eyes, instead of having to
 stare down at our phones.
 But the glasses and headsets that have come out have either been awkward or have found
 a niche firmly in the enterprise market, which is what a company called Avagant thinks it
 can change.
 This is a mixed reality headset.
 It's called mixed reality because you can still see the physical world around you, but
 you're also seeing digital objects laid over the real world.
 And this is Avagant's newest headset prototype, the one that the company has built to show
 hardware partners what it can do, specifically with light-field displays.
 Now like Microsoft HoloLens, the Avagant headset shows digital objects that appear to be floating
 in space.
 Unlike HoloLens, this headset prototype is tethered to a PC, running Unity.
 This is the optics engine that we have.
 This is what, you know, creates the entire image.
 This has our retinal imaging technology, the same technology we've used previously in
 the Glyph.
 So we're using micro-mirror technology to project images to your retina, and also inside
 of this we've incorporated our new light-field optics in here.
 We're simultaneously sending you multiple focal planes.
 What that means is we're not actually tracking your eyes, which means you can even do this
 with a single eye.
 So if you close one eye, you can feel the focus change with a single eye between the
 two points.
 So what can you do with something like the Avagant headset?
 Well, I navigated the solar system and saw the room around me transformed into a giant
 fish tank.
 The experience was partly enabled by camera tracking systems, placed either in the corners
 of the room or directly on the headset.
 I also tried a couple of apps that have been shown in promotional videos but haven't been
 experienced by many people in real life before.
 One app was a beating human heart, and it was originally made for HoloLens, and another
 app let me hold digital objects right in the palm of my hand.
 Alligator.
 Rhino.
 Oh, a syringe.
 Oh, that's creepy.
 Ew, it's going into my hand.
 Oh, I have a syringe stuck in my hand.
 This is bizarre.
 Okay, so in order to get a sense of what Avagant is trying to do, it helps to understand
 how light field technology works, and it comes down to how we perceive depth.
 If you hold something up in front of your face, for example, what you're seeing is
 the light that bounces off its surface and passes into your eyes from multiple angles.
 Your brain combines the 2D image from each eye and uses visual cues like the rotation
 and focus of your eyes to create a sense of depth.
 You'll also notice that you can really easily shift your focus from the thing that's right
 in front of your face to the stuff that's in the background.
 When you start looking at digital images through smart glasses or mixed reality headsets, you're
 usually seeing a flat 2D image with a fixed focal point.
 And when you've got a display mashed right up against your face, it just doesn't feel
 natural.
 By using light field optics in its headset, or what Avagant calls its multifocal plane
 approach, Avagant says it's essentially cutting up the images in front of you into different
 focal planes and sending that light into your eyes.
 So when you go to look at different digital objects, it's supposed to replicate the way
 your eyes would naturally focus without a headset on.
 So right now as I'm looking at you, you're in focus, but the arcade game behind you is
 not.
 But if I were to keep moving my eyes back and forth, they're sort of naturally adjusting.
 And you're saying that when someone wears an Avagant headset, that your technology lets
 that happen sort of naturally.
 It doesn't feel like you're straining your eyes.
 That's right.
 So when you try the light field technology, it should not only feel more realistic, it
 also should feel a lot more comfortable and natural for your eyes.
 There are some elements of Avagant's technology that the company declined to talk about, like
 the number of planes it uses in this multifocal plane approach, and exactly how it's creating
 this continuous focal experience.
 One industry expert pointed out to me that terms like light fields and holograms can
 be a little overused.
 Generally speaking, the best way to look at these kinds of solutions is to consider them
 a practical approximation of an analog light field display.
 Right now, Avagant seems confident that its technology is superior to at least some of
 its competitors out there.
 Although it's hard to say how it stacks up to other companies that are holding their
 tech pretty close to the vest.
 So how is that different from what, something like a HoloLens?
 They're a fixed focus, right?
 So the image that you see is fixed at a certain distance.
 Now they try to use stereoscopic 3D effects to make your eyes converge at different angles
 to give you a sense of 3D, but they're not really changing the focus of that image.
 That's okay when things are generally farther away, but that whole display and experience
 breaks down the second that you get within about a meter.
 So you think about, if I want to hold an object in my hands, if I want to have a face-to-face
 conversation with a virtual person, you actually really can't do any of those things unless
 you have a light field display, unless you solve those focus problems.
 How does it compare to what Magic Leap is doing?
 We don't know exactly what Magic Leap is doing.
 Have you seen Magic Leap?
 No, I haven't.
 Maybe you guys have a better idea.
 I'm sure you've read a lot about it.
 Certainly.
 I mean, we can just see what's in the news.
 We don't know exactly what they're doing, but at least from what's public, I would say
 that we both believe that light field is a critical part of the experience.
 You might have heard of Magic Leap by now.
 It's a super secretive company in Florida that has raised more than a billion dollars
 to date to finance its digital light field technology.
 I reached out to Magic Leap and not surprisingly, the company didn't want to share specific
 details on their technology.
 But one thing that Magic Leap has said is that they're building what you might call
 a full stack solution.
 They're building the computing system, the headset, the software.
 They're even working with content creators.
 Avagon's plan for now isn't to build a product it's going to ship directly to consumers,
 but to partner with other hardware companies who will use Avagon's optics in their own
 headsets.
 This could come as soon as 2018, Avagon says, but it could take longer to get to the consumer
 market.
 But when that happens, they believe there's a very real chance that something like this
 could replace the smartphone as we know it.
 We always knew that as things were moving to mobile devices like phones and tablets,
 that the next evolution with that was going to this idea of the ambient computer.
 Something that I can put on like glasses and wear and suddenly I'm surrounded by these
 experiences and not limited to the screens that we can fit in our pockets or fit in our
 bags.
 So do you really envision a world in which people are just walking around wearing air
 glasses all the time, even if they're not currently people who wear glasses or something
 on their face?
 I actually do.
 I think the smartphone as we know it today will get replaced by smart glasses.
 And once the industry overcomes some of the challenges that still exist today to make
 the devices much lighter and less bulky, I think yes, people will be wearing those glasses
 for an extended period of time.
 This is the thing that a lot of technologists are talking about right now.
 What's the next big thing beyond smartphones?
 Is there a thing beyond mobile or will we all be staring at the glass rectangles in
 our hands for many more years to come?
 If we really are approaching a post-mobile world, then Avagon is trying to position itself
 at the forefront of this.
 And while a lot of headsets so far haven't really caught on with the mass market, Avagon
 thinks there's a time in the very near future when these things are just like eyeglasses,
 that you just carry them in your pocket and slip them on instead of staring at your smartphone.
 It's a bold idea of a computing future, and it's going to take a lot more than optics
 to make it work, things like apps and content that make people want to wear displays on
 their faces.
 It might be a future that we're not quite ready for, but it might also be one that's
 happening anyway.
 You must never get tired of looking at people while they wear this thing.
 You get very used to it.
 You know, it's the typical look, you sit there, you're very quiet, jaws always open a little
 bit, just like you saw for the first time.
 What's funny is I actually can't hear you at all right now.
 I just see this.