Augmented Reality: Bridging the gap between the physical and the digital world
preip
There has been a lot of talk about Virtual Reality (VR), but still there are very little applications to enhance our everyday lives outside of entertainment. Augmented Reality (AR), the less known sibling of VR, has the power to have a more profound impact on our lives than VR ever could. Instead of replacing the real world with a virtual one, AR enhances the reality with virtual content. Therefore, AR can be a gateway for people in accessing and understanding todays technology and could provide vast possibilities to support our everyday lives, e.g., for navigation, traveling, or education. This talk will give an overview on AR in general and explain its possible benefits and use cases, as well as the issues that may arise, e.g., regarding privacy, data security, as well as psychological and sociological challenges. The talk requires no special knowledge and is suited for people with little exposure to AR and mixed reality, but it will also give insights into current relevant research and development.
 We're going to look at how to bridge the gap between the physical and the digital world. So we go into immersive things. And my guest here, our guest here is Patrick, and he goes as well with the nickname, Prype. He's currently doing a PhD on human-computer relations interaction. I'm not going to discuss this since I'm a little bit on this field. We made an agreement about that. But he's focusing on augmented reality today. He's doing this study in Dresden, I understood. He has a general interest of course in things to do with 3D. That goes along with these kind of things. And his talk provides us an overview on AR in general and explains its possible uses for goods and for the bad. Ladies and gentlemen, here is Patrick. Please give him a warm welcome. Thank you. Like the views and put it in play, Patrick. Thanks. Hi everyone. This is a foundation talk. It's mostly aimed at people who know nothing about augmented reality. So, hopefully after the talk you will know a little bit more. However, if you have some experience, I hope maybe you will get some new impulses. I don't know. Furthermore, the talk is also focused more about how augmented reality can be used for everyday life and not so much on the scientific community. And it's roughly 30 minutes or something like that. So, I made a little shorter so that we have some questions in the end. So, if you have something that's of particular interest, just ask afterwards. So, but first let's talk a little bit about virtual reality because I think most people maybe are more familiar with the term. There has been a lot of talk about virtual reality like changing and revolutionizing the way we use computers. There are as many commercial virtual reality assets as ever before and also quite a lot of applications. But somehow we are doesn't really influence our everyday lives that much. And I think there's two reasons for that. The first one is that virtual reality applications are mostly centered on entertainment right now. And the second is that it disconnects us pretty much from the real world. So, why it might have the ability to actually transfer us into another world which more or less looks like this. At the same time we are cut up from reality and in contrast augmented reality expands the real world. And in the mind of most people right now this looks something like that. But the important thing is that we can still perceive our surroundings and we can also experience the virtual content as well. And this particular difference makes AR a lot more useful for everyday use than virtual reality. So let's go to a little definition of what augmented reality actually means. So the important thing is that we combine real and virtual content, blowing the borders between both. So as you can see here's a virtual globe on a real table. And the application also interacts in real time. So if I move or if I have some input, the application reacts on the spot. There's no delay. And the objects are situated not relative to the user but in space. So if I move around the globe stays on the table. So this is basically what I'm talking about augmented reality, what I mean. But what is also important to know that AR is only part of the so-called mixed reality continuum, which ranges from reality without any augmentations. So the normal reality we experience every day to a completely virtual environment. And those four categories are basically the classic mixed reality continuum. There's also some extensions to that, which you can see here. I don't want to go into any detail of what those all mean. But the important thing to remember is that there is no hard border where augmented reality begins and where it ends. There's also a lot of ways to provide augmented reality. Mobile augmented reality like Pokemon Go, where the AR content is perceived to a mobile device like a smartphone, is the most common form today. There's also projected augmented reality, where you have a projector superimpose an image onto a real environment, which is especially useful if you have like collaborative scenarios where the content should be perceived and manipulated by a group of people at the same time. You can actually see this example right out of the door. There's a similar installation there. And then there's what this talk is also mostly focused about, head-mounted augmented reality, where users wear AR glasses, which can be seen through, and where the virtual content is projected onto. And this is the most immersive form of augmented reality. And when you have sophisticated technology, it's more or less indistinguishable from the real world. So that's basically the goal where we want to get. So, but now it's the question, OK, why augmented reality? What's so important? Why do I have to talk at all? And yeah, for that, I want to quote Tim Sweeney of the CEO of Epic Games, because it's always a good idea to quote someone who's more famous than oneself. And he said that he believes that augmented reality will be the biggest technological revolution in our lifetime, because once you have an AR display, you don't need any other form of display. You just take your screen wherever you go. So that's a very powerful thing to say. You also think that the revolution will happen in the next 10 years, which is very optimistic from my perspective when you look at what the current technology can do. But nonetheless, I believe this revolution is going to happen sooner or later. So it's useful to familiarize yourself with the term. What is also important to note is that the goal is not to just substitute a real display with virtual ones, but augmented physical objects themselves, that the augmentations really have a relation with the location they are situated in. So only then AR will have really the power to revolutionize our everyday lives. So I will elaborate on that a little later on. But first, a question like how likely are fundamental changes in our society triggered by augmented reality? So if we compare it with smartphones and the way they are used, I think it's very likely. Because smartphones today are technology focused directly on supporting the way people live their lives every day. So it also started as a research a long time ago, before commercial products were available, and now they are ubiquitous, they are an integral part of people's lives, and for some people it might be difficult to actually live without them. And I think when AR is likely to do the same, when it's focused on the things people actually need, and once the technology comes along. But before we go there, a short, very short history of augmented reality so that you can see where this is all coming from. So actually augmented reality is rather old. So the first augmented reality, virtual reality device, was built by Ivan Sutherland in 1968. It was called sort of a thermoclass, but it had already had tracking and see-through optics. So that's quite advanced for the time, I would say. The actual term augmented reality was only coined 25 years later by Kauder and Mitzer in 1992. So quite a long time in between, but of course the research still stood not during that period. Here's an interesting example that I wanted to show you from 1994 by Stade et al. which shows an ultrasonic image of a human fetus directly superimposed over a woman's belly. And yet it shows that even very early on that people were thinking about how we can use augmented reality to really benefit humans in a lot of ways, despite the technology at that time being very limited. This is also an important step because it was the first mobile AR headset for usage outside of the lab with a wonderful name Turing Machine by Fine Art Art in 1997. So even like early on the question was okay, how can we use this technology not only indoors, but how can we use it outdoors as well, which is very important. And then finally the first AR application using mobile devices, in this case PDAs for the people who still remember them, with extended cameras. And you can see those little markers in the background which are tracked by the cameras and then little virtual trains are overlaid over the wooden train tracks. So it's not really that practical just to like a proof of concept more or less. So but yeah, like I said mobile AR today is the most common form of augmented reality, but maybe that might change soon because now we have like the currently the first generation of really commercially available augmented reality headsets and therefore also an increased interest from the scientific community, but also from the industry for solving their various problems. So this immediately brings us to the areas of application and hopefully how augmented reality may influence our lives in a positive way. First a little bit about the research. Actually only 12.5 percent of the research is actually focused on application cases for augmented reality based, the statistics are based on the international symposium on augmented reality. And the rest is focused on things like tracking technology, interaction, rendering and so on. And of those application cases most of them are usually industry centered and not so much on everyday life. So we have things like for example logistics, you have things in the medical sectors and of course they have a great indirect influence on our lives, we will still benefit from those things, but I want to focus a little bit more on everyday use, so how can we immediately benefit from augmented reality and explore that a little further. And those are things like guides and cultural heritage for example and tailor communication and tailor presence, which also brings us to the first use case which is AR tailor presence, which is more or less like video phoning only more with 3D avatars of the person you're speaking with embedded directly into the environment. So basically like a virtual avatar sitting on the chair next to you, which is I would say rather cool because it's something that's out of sci-fi novels or films, but now it's here. Now we can use it and despite being far from perfect, it's still very incredible, at least from my perspective. And you can not only have a conversation in real size, but you can also scale it up and down as much as you want, like in the lower right image you can see that the girl is standing on the pedestal in front of the guy, very small in size, you can actually have people, virtual people running around your table where you do other stuff while talking to them. So another very classic example I would say is AR navigation. This is a system by Google, which is a direct advancement over their smartphone navigation. And I think despite being a very obvious example of when you think what AR can do for you, it's also very powerful one. So basically it means like when you want to go somewhere, you project like for example a line in front of the people that you have to follow to reach the destination. It's very easy and very understandable. Currently there are mostly specialized systems that focus on something like airport navigation or car navigation for example, very few for general pedestrian use, but I think that's likely going to change in the future. So the next one also maybe a little obvious finding places of interest based on selected preferences. So the example would be like where's the next post box, where's a good restaurant or bar. So this is of course possible today with smartphones as well, but the big difference is that you don't have any abstraction. So you don't have to match the map with what you're seeing. So where is this building that it says on my smartphone. You get the information placed exactly at the point of interest itself. So that makes it a lot of easier and useful for example. Of course you can also show something like user reviews or for example the menu of the restaurant so that you can find a suitable match. What's also very, very useful was something like AR translation, where you like provide an instant translation of a foreign language in place. So for example, you look at a sign in Russian and you get the content in English. This is actually Google translate, which maybe some people already have used. Yeah, what's like you would when you have used it, you would say, oh, well, that's really practical for example. And when you use it with a head mounted device, it's even more practical because you don't actually have to point a device somewhere to just see the text and the language you prefer. So but besides translation, you can also offer context information as well of course, because you can not only show in this example the direction, but you can also show the distance to the object. And it's also suitable for places like museums, where space for text is usually very limited, but space in AR generally is not. Which brings us to the next application case, which is cultural heritage and tourism. And this is a nice paper that gives a good example of that because it can make history somewhat more experienceable. So when you have ruins, for example, of a Greek temple like here, you can get an impression of what the temple might have looked like directly situated on the actual ruins. Or when you have the fields of antique running tracks, you may have a virtual efflates on augmented reality that give you an impression how these facilities were actually used, like the people running around doing sports and stuff like that. And I think this is really powerful. I think you just get to see what it was looked like in the past. But another important aspect, and I think a very powerful one, is how AR may have the power to make technology more accessible again. So today's technology grows more complex and it's more and more connected. So that's even for experts, it's sometimes hard to actually say, okay, what state is the system currently in? For a layman person, it's even harder. So AR can help make this hidden information visible again in a way that's easy to understand. So just to give you a little example, which is not perfect, but hopefully sufficient to transport the idea, imagine you have a simple router at home, for example. You have some status LEDs, but for whatever reason, your internet breaks down, stops working, they don't have you that much for debugging the actual problem. So to get more information, you have to most likely access a web interface if you still can remember the IP of your device. Of course, you can do what we have done here and put a little piece of paper with the IP on the device itself, but that's hardly a practical solution. Yeah, so imagine with augmented reality, you can just look at the router, do a simple gesture, and then get the information you were interested in directly superimposed on the physical device itself, which can help you immediately recognize what the problem might be. So maybe you just forgot to turn on DHCP or something like that. But we can expand that concept to not only include a router, but also the wireless devices connected to it. So one of the predominant features of wireless LAN is that there isn't a wire, so it's hard to see which devices are connected, where they are located, and what the IP is, and so on. So where we are, we can simply draw virtual connections to make that information easily available to the user. So I do something like this. But we can also use it to make errors more recognizable. For example, one of the devices may be not be able to resolve websites because there is no DNS server registered. And then we can, of course, also display additional information like the bandwidth the device uses, and when we have an actual physical medium, we could actually use that for displaying information like a graph of the bandwidth of a time. And it's very easy to understand what is displayed because it's directly located at the medium itself. Yeah, so I hope this example gives you an impression of how AR can really help with understanding technology again. We might talk a little bit about that later on. But first, let's come to some technical challenges. So how does that actually work, and what has to change so that we can really use it every day? The first one is tracking, which is one of the main challenges, because I have to locate the user somehow. I have to know where he or she is and what she's looking at, for example. So for general positioning, I could use something like GPS, but precision is, of course, a problem, and I can't really use it indoors. So we need additional tracking technology. And there are many different approaches, but the current state of the art is more or less infrared tracking with time of flight. So that means infrared light is emitted, which is reflected by the environment, and the time between sending and receiving the light impulse determines the distance to an object. And with that, I can create a depth model of a room, which more or less looks like this. And then I can not only tell where the user is right now, but also when he or she is moving around. And while this technology is far from perfect, I think the current tracking technology is quite robust and useful in that regard. What's also important is display and rendering. So there are many different technologies for head mounted devices. One of them is video see-through, where camera films the environment, which is then together with the virtual content rendered onto a display in front of the USSR. But latency is a very big problem in that regard, because you will very easily get sick. Another is see-through optics using polarized glass, where the air content is projected onto. And there are even small laser projectors, which project directly into the user's eyes. But for some reason, user acceptance is not very high. But the ideal display would not require any external display at all, but you would directly manipulate the optic nerve. Which sends the messages to the brain. Well, we don't have that right now, so we have to make through what we have. And this is basically see-through optics, which is right now like the current state of the art. So let's look at what the problems are in that regard. One of the problems is opacity, because when I have a solid object, I have a solid object. Because when I have a solid object, like a human for example, no background should bleed through them. So currently for the commercial headsets, this is not so bad. But really depending on the lighting conditions, when you have a lot of sunlight, you can't really see anything anymore. But yeah, there's still much room for improvement in that regard. What might be even more important is the field of view of the AR headset, because like a human has roughly 210 degrees field of view. And as you can see here, like the Microsoft HoloLens has like 30 degrees, and the Meta 2 has like 90 degrees, most of which for my experience at least are pretty blurry. So yeah, that's not... There still has to get a lot better until we can really use it. Also very important, when a person wearing an HMD, what does she see? Like how can I recognize what a person is looking at? And how can I get data on the recognized objects? So due to performance requirements, most of the processing will most likely not happen on the head mounted device, but it will very likely take the form of some web service or similar services. Which not only process a constant visual input stream of the cameras of the HMD, but also have to provide data sources for the various objects. For example, when I have touristic points of interest, I need to get some information on them. So this requires a constant data stream and a very good connectivity to make it really useful. And it also requires a lot of services readily available to use by the AR device. So this is quite a big challenge, maybe the biggest of them all. So to give you a little impression how that could work based on existing research, which is called a reality-based information retrieval. So imagine you have an AR HMD that feeds a constant live image to a server to recognize objects the user is currently looking at, for example, vegetables at a market. So the server is able to classify those images by comparing them with a couple of saved reference images and gives a list of recognized objects and the confidence value back to the HMD. And then the HMD can query a different servers to get some recommendations for meals that I can cook with that vegetable. So when I explore a market and you have your AR display with you, you could not only tell you what a particular vegetable is in case you might not know, but it could also tell you what to do with it, which more or less could be quite useful. What's also important, like, if AR glasses are to become everyday objects, economics and aesthetics are going to be important. So you probably don't want to wear something like this right now. So for the most part, that means shrinking the glasses down to the size of current normal glasses, which was a form factor that's more or less already accepted in our society. And as you can see, in the comparison, we are not really close to being at that point. And one of the most important questions is not only how to display the AR content, but also how to interact with the content, because otherwise it's a very limited experience. So our AR objects might appear to be part of our environment. We can't really touch them. So I can't actually press a virtual button. And there is research concerned with providing haptics as well. For example, there is some ultra haptics, which uses ultrasound to create resistance in midair, so that I have an actual resistance when I touch somewhere, but it's still in a very early state. So currently, mostly gesture interfaces are used, where my fingers are tracked, and the line of sight together with the hand gestures are used to trigger interaction. Speech input is also a common solution, but use in public environments is limited. And another possibility is also to use additional devices like specialized controllers, or for example, the touchscreen of a smartphone for interacting with the virtual content. Gaze tracking, that means tracking the user's eyes, will most likely also play an important role. But right now, there is pretty much no perfect solution, and also still a lot of research happens in this regard. So this is a very hot research topic right now. So let's come to possible dangers and other issues. So all the things that might not be so great about ubiquitous augmented reality. The first one, pretty obvious problem, is information overload, where too much information leads to excessive cognitive load by users. So virtual content has to be used sparsely to best support the users, and the only way to make sure users are not overstrained is to leave them in control of what and how much is displayed. But users might not always have the means to control it. So especially when third party services are unwarfed, we may see something like invasive advertisement, sorry, and other unwanted content. And since head-mounted AR is already a personal experience, these ads are most likely to be personalized as well. So this directly leads to the question of privacy and data security. There will most likely be a constant stream of the user's current position and a live video feed to various third party services. So this raises the old question, how can I control what happens with my data and who gets access to it? And privacy, of course, is also a big issue, because there's hardly anything more personal than a constant live feed of what I'm seeing. So even if I myself decide to don't use an AR device, as long as everyone else does, it hardly makes a difference. So at the center maybe is really the question of who controls the actual hard on software platforms. So similar to smartphones, it's likely for a few global operating corporations to not only offer the AR devices, but also provide the corresponding software ecosystems as well. So we see that already with the direction the HoloLens by Microsoft is taking. But AR consists of even more sensitive data than current smartphones. So will we be in control of our own devices? Do the companies producing them have any interest to keep us in control? Because most likely their primary motivation is to make money. And will people even care? Because they hardly seem to care with smartphones right now. So I think this is an important question. And another possible application area for augmented reality, which I haven't talked about, is military use. So heads up displays have been used in the military in aeroplanes, for example, for over a decade. And there also have been several scientific papers evaluating the use of augmented reality in military use. So it's not difficult to imagine AR systems like telling soldiers where to go and maybe even who's friend or foe. Microsoft has recently made a contract with the US military for over 480 million US dollars to evaluate the use of the HoloLens on the battlefield and the goal is to increase lethality, mobility and situational awareness of soldiers on the battlefield. So I don't really want to provide any judgment on that, but it's best to not have any illusions what this technology can and eventually will be used for. And it's probably not all going to be unicorns and rainbows. So let's keep that in mind. Yeah, another interesting aspect is how does massive use of augmented reality actually change how humans interact with each other? So for example, when we can project ourselves everywhere at any time, how important will it be to meet in person? So to give you an impression, there's the novel Naked Sun written by Isaac Asimov in 1957, where it's common for people to communicate with photorealistic holograms and even being naked in front of each other. There's no problem, but it's extremely unpleasant for people to be in the same room with a real person. So I'm not going to say this is what is likely to happen, but I think it's an interesting aspect to think about as there will be some social changes involved, which we don't know yet. This also leads to the question how our own way of thinking is influenced by the constant use of augmented reality. So for example, how does our perception of locations and how locations are connected change when we are guided by AR navigation, which in the worst case means following a line until we reach our destination? There has been research showing that extensive usage of GPS might lead to an atrophy of the hippocampus, increasing the risks of cognitive diseases like Alzheimer's later on in life. So it's important to be aware that there might be issues and it's important to do the actual research to figure out what these issues might be. And of course, maybe even the ultimate question, what is reality? Like how does our understanding and perception of reality itself change when we use AR? This is a metaphysical question, but I think it's an interesting one nonetheless. It's not so much a problem for moderate use of augmented reality, for example, to annotate objects, but more a question for photo realistic augmented reality, which can alter the way the world looks to each person individually. So if my world looks vastly different from your world and when AR is always active, what does reality really mean in this regard? And what role, if any, does the unaugmented world play in the lives of people anymore? So this might be something to think about. And with that, I conclude my talk. So hopefully I could show you that augmented reality could have a benefit in our life, but that there are also some risks involved and I hope you found it somewhat interesting. And now we can have some questions. Thank you. Thank you for your fantastic talk, good overview of whatever is possible. I see some people here queuing to ask you some questions. Let's go to number two here. Please, sir. Thanks for the talk. The previous slide on the red and blue pill. Can you elaborate a little bit on how you as a researcher and user of augmented reality eventually better understand reality? So you mean when we use augmented reality that might lead to a little better understanding of what reality is or what did you mean? Yes, that's the question. Do you feel like you're not just using but also researching because in order to, for example, when you have this slide on how the 3D scan works, you use it as, let's say, a metaphor or a shortcut on how our own brain and visual system works? Yeah. Well, like I said, it's a rather metaphysical question. So it's hard to really answer the question right now. It's more like you could guess what it might, how it might change for people, what the perception of reality, how does that change? But I don't, I can't really answer that, to be honest. Like it's more of an open question for myself as well because like when I think about augmented reality and if it eventually will be used by everybody every day, like I can think of some like dystopian sci-fi scenarios where everyone lives in this happy world, which is totally fake. And the real world basically looks awful, but nobody actually noticed that because everyone is like caged in his own happy place, for example. But I don't know if that's going to happen. It's just a possible danger that you could think of. But this is probably something that will not happen in the next 20 to 30 years because we simply don't have the technology to do that right now. So maybe that more or less answers your question, I hope. Yeah, next one. Yeah, I was thinking along as well. Okay. Sorry. So to get widespread consumer adoption, we really need the miniaturization of the hardware. Google Glass got kind of laughed out of the room a few years ago, so we need to be better than that. Yeah, but Google Glass wasn't really augmented reality. It was more like a headset display because you had this tiny screen in the back of the upper part of your field of vision. And it was like you could just display a limited amount of information there. Like when you have real augmented reality, it's basically having the whole space to put information. Sure. I mean more like it's a stupid looking thing to walk around on your head. So we've got to look better than that when we're for the mass consumer adoption. But that's why I think that we have to go to where medical glasses are right now, because they are already accepted in society. So someone wearing glasses doesn't get looked at in a funny way because it's pretty normal. So if augmented reality headsets look more or less the same, I think we're fine. So the question is what are the technological problems that we have to solve to get there in a consumer device? Yeah, we have to actually miniaturize the stuff. So I think the laser projectors actually are an interesting solution because they are rather small and you can project directly into the eyes of the users. So if we get this user bias figured out, how we can actually convince people that this is a good idea, not dangerous at all, we can shrink it down even further. And if we have the whole processing, not on the device itself, but on third party, like data centers or something like that, we can also shrink that down so we have to actually only stream the input to the server and we have to stream the image back to the device. Now we can build pretty small devices, I think. So that's what we have to figure out. But yeah, I think this is something that's actually going to happen in the next five years or something like that, because this is actually something that's worked on right now. Microsoft actually is building the next version of HoloLens, which should be smaller, which should have more features like eye tracking and stuff like that. But we have to wait and see, basically. This is also an important thing. The hardware involved is very complex. So it's not something that one person can do in his garage or something like that. So you pretty much only have the chance to have those big corporations do this stuff and do the research because it's so complex. With a little bit of imagination, we see someone stepping in there from the Internet who has a question. Olographically projected here next to me on stage. The Internet has a related question. How long do you think will it take until everyone uses augmented reality like mobile phones today? I think that really depends on the use cases. I think this is the most important thing for acceptance. You have to provide functionality that actually benefits the way people live their lives. For example, the R translation is a good example because it's common for people when they are travelling, for example, to run into signs or something like that, which they can't understand because it's a foreign language. And if you provide an augmented reality solution for this particular problem, people are going to use it because it's practical, because it's useful. And the more functionality, it's the same with smartphones, really. The more functionality you provide that goes in this direction, the easier the adoption will be. So yeah, that's basically, we have to find use cases which are important to people and then the adoption will basically happen itself. Critical mouse, probably. Yeah, critical mouse. Number three, please, sir. Thank you for your talk. When you introduce the platforms, what is with Facebook with the Oculus Rift? Is it not virtual reality? Yeah, it's virtual reality. It's not really augmented reality. Yeah, I was thinking about including it, but I still don't have any idea where they are going with this. So they have bought this company. They have invested in this technology. But yeah, I don't see where they are going with this right now, what their goal is actually. So this is why I didn't include it. Thank you. Yeah, it will come in your direction anyhow. But number two, please. First of all, thank you for the informative talk. And what do you think will change from privacy policy with when you have an active life feed of what you're seeing with augmented reality? Yeah, I think like this may be the cynical way of looking at it, but I think people will stop caring about that. That is what simply is going to happen. Like the current generation is concerned with privacy, and we feel like for us it's a very bad idea to actually have a constant life feed of what we are seeing transmitted to somewhere on the internet. But I see that with younger people who are smart phones. They don't think about privacy pretty much. They think about what this device can do for them. And I think this is like more or less the social change. Maybe that will happen. People will stop caring about it. It will just be a normal thing that your field of vision gets transmitted to somewhere on the internet, and people will simply stop care. I think that is what's going to happen here. Thank you. Okay, next number two. Thanks. I saw recently there was a company that announced some shades, and they had bone conducting sound transmission. So they didn't really project something into the retina, so they went away around all the projection and all the complexity of making this huge bulky headset. Do you think this is a viable way to go forward? You've focused mostly on visual inputs, but maybe other sensors are better suited to making an entry into augmented reality. What do you think about that? An interesting point. I focus a lot on the visual side of things, but the important thing, I guess, with all this mixed reality and virtual reality stuff, is that it's not limited to the visual. So basically it should be a whole sensory experience, so all your senses are involved. So yeah, maybe that's a good idea to not only focus on the visuals so much, but also maybe on things like audio, like you said. It's really the same question with the use cases. When you figure out a very good use case where an audio augmentation helps people in their everyday life, I think people will start to use it. But the reason why I focus mostly on the visual part is because I think it's the most strongest sensory experience people usually have, one day have eyesight at least. And I think this is most likely also to be like the selling point of augmented reality, where most likely to be the visual part. Okay, another one. Number two. Thanks for the talk. You mentioned that one of the ways you see the miniaturization of these devices evolving is by streaming data into the cloud. But we've seen, I mean, especially with smartphones these days having the equivalent computational power of computers from a decade ago, that it is actually possible to do a lot of these machine vision tasks on a smartphone. And I think some manufacturers are basically trying to get your smartphone to be your primary computer. So do you see that as a likely alternative? Because obviously this comes back to the privacy question because if you can keep things local. Yeah, but it also comes back to what the producers of the devices want you to actually do with the devices. Yeah, of course, when you have miniaturization it is powerful enough to put a whole computer into something like normal glasses. You could do that and you don't have to stream that much information. But I guess what you still have to actually do all this stuff because you are not alone in the world basically with augmented reality. So if everyone else is using augmented reality you need to have that actual data from what the others are doing. Like you have to build, for example, in this building, I have to build a constant room model of how this building looks like. So everyone who is moving through this building will constantly scan the environment to build the 3D model of the environment. But you have to share that information so that when I move into this building for the first time I still have the whole environment mapped out because someone else already walked through it. So you still have to stream all that and you still have to share all those data. So I think even if you increase the processing power of the devices itself, this is not going to change much. Okay, sir. Okay, my question is very much related. With all the big players designing the devices, it's still a closed world. And I would have to imagine I would have to stream my live feed to some Microsoft server or whatever. Are there any efforts to keep at least interfaces open if not the whole software open source? So for example, the CCC could put up a server I trust and send my live feed to to get a process. The system is open or not that you can actually change the location where your information is transmitted to. Yeah, well, at least I don't know of any open hardware, open software projects who try to do augmented reality right now. Of course, there's research. Research is more or less open most of the time, but research prototypes right now, while they are good at what they do, they are hardly like consumer products. So you can't really mass produce them and distribute them to the people. So yeah, you probably could do it, but I don't see currently I don't see any movement in this regard. Yeah. I shouldn't do that, but just for the previous question, I would suggest. Speak up, please. I shouldn't do that, but to answer briefly the previous question, WebAR and some effort from Mozilla are trying to bring at least the highest level of the stack in an open source fashion for augmented reality. Okay. Yeah, I was wondering for the example you showed on your research in your own lab about information retrieval. Those examples that are able to distinguish between one object and another, let's say an apple and a pear or a tomato works well, but I'm wondering how scalable it is because when you get to a morphicized example. Yeah, that's a good point because like what we were using was an actual food database for that. So it was very easy to classify like the vegetables because they are food. But if you have a database with everything in it, of course, it's a problem because like classification will get you some weird results. So maybe the vegetable would be recognized. I don't know as a lamp or something like that. Yeah, that's that's a problem. But this is only a problem like in this particular research case, it was more like a proof of concept to actually show. Okay, how can we use augmented reality in some more practical everyday scenarios. It wasn't really much focused on to find the perfect technical solution for providing like the classification of the images. This is also why we use an existing service for that and we didn't build our own. So we just use what was available. Okay, unless someone is beaming himself up from the Internet, then we had all the questions here answered. Thank you very, very much. Thank you. I hope to see you soon.
 We're going to look at how to bridge the gap between the physical and the digital world.  So we go into immersive things. And my guest here, our guest here is Patrick, and he goes  as well with the nickname, Prype. He's currently doing a PhD on human-computer relations interaction.  I'm not going to discuss this since I'm a little bit on this field. We made an agreement about that.  But he's focusing on augmented reality today. He's doing this study in Dresden, I understood.  He has a general interest of course in things to do with 3D. That goes along with these kind of things.  And his talk provides us an overview on AR in general and explains its possible uses for goods and for the bad.  Ladies and gentlemen, here is Patrick. Please give him a warm welcome.  Thank you. Like the views and put it in play, Patrick. Thanks.  Hi everyone. This is a foundation talk. It's mostly aimed at people who know nothing about augmented reality.  So, hopefully after the talk you will know a little bit more.  However, if you have some experience, I hope maybe you will get some new impulses.  I don't know. Furthermore, the talk is also focused more about how augmented reality can be used for everyday life and not so much on the scientific community.  And it's roughly 30 minutes or something like that. So, I made a little shorter so that we have some questions in the end.  So, if you have something that's of particular interest, just ask afterwards.  So, but first let's talk a little bit about virtual reality because I think most people maybe are more familiar with the term.  There has been a lot of talk about virtual reality like changing and revolutionizing the way we use computers.  There are as many commercial virtual reality assets as ever before and also quite a lot of applications.  But somehow we are doesn't really influence our everyday lives that much.  And I think there's two reasons for that. The first one is that virtual reality applications are mostly centered on entertainment right now.  And the second is that it disconnects us pretty much from the real world.  So, why it might have the ability to actually transfer us into another world which more or less looks like this.  At the same time we are cut up from reality and in contrast augmented reality expands the real world.  And in the mind of most people right now this looks something like that.  But the important thing is that we can still perceive our surroundings and we can also experience the virtual content as well.  And this particular difference makes AR a lot more useful for everyday use than virtual reality.  So let's go to a little definition of what augmented reality actually means.  So the important thing is that we combine real and virtual content,  blowing the borders between both. So as you can see here's a virtual globe on a real table.  And the application also interacts in real time.  So if I move or if I have some input, the application reacts on the spot. There's no delay.  And the objects are situated not relative to the user but in space.  So if I move around the globe stays on the table.  So this is basically what I'm talking about augmented reality, what I mean.  But what is also important to know that AR is only part of the so-called mixed reality continuum,  which ranges from reality without any augmentations.  So the normal reality we experience every day to a completely virtual environment.  And those four categories are basically the classic mixed reality continuum.  There's also some extensions to that, which you can see here.  I don't want to go into any detail of what those all mean.  But the important thing to remember is that there is no hard border where augmented reality begins and where it ends.  There's also a lot of ways to provide augmented reality.  Mobile augmented reality like Pokemon Go, where the AR content is perceived to a mobile device like a smartphone,  is the most common form today.  There's also projected augmented reality, where you have a projector superimpose an image onto a real environment,  which is especially useful if you have like collaborative scenarios where the content  should be perceived and manipulated by a group of people at the same time.  You can actually see this example right out of the door.  There's a similar installation there.  And then there's what this talk is also mostly focused about, head-mounted augmented reality,  where users wear AR glasses, which can be seen through, and where the virtual content is projected onto.  And this is the most immersive form of augmented reality.  And when you have sophisticated technology, it's more or less indistinguishable from the real world.  So that's basically the goal where we want to get.  So, but now it's the question, OK, why augmented reality?  What's so important? Why do I have to talk at all?  And yeah, for that, I want to quote Tim Sweeney of the CEO of Epic Games, because it's always a good idea to quote someone who's more famous than oneself.  And he said that he believes that augmented reality will be the biggest technological revolution in our lifetime,  because once you have an AR display, you don't need any other form of display.  You just take your screen wherever you go.  So that's a very powerful thing to say.  You also think that the revolution will happen in the next 10 years,  which is very optimistic from my perspective when you look at what the current technology can do.  But nonetheless, I believe this revolution is going to happen sooner or later.  So it's useful to familiarize yourself with the term.  What is also important to note is that the goal is not to just substitute a real display with virtual ones,  but augmented physical objects themselves, that the augmentations really have a relation with the location they are situated in.  So only then AR will have really the power to revolutionize our everyday lives.  So I will elaborate on that a little later on.  But first, a question like how likely are fundamental changes in our society triggered by augmented reality?  So if we compare it with smartphones and the way they are used, I think it's very likely.  Because smartphones today are technology focused directly on supporting the way people live their lives every day.  So it also started as a research a long time ago, before commercial products were available,  and now they are ubiquitous, they are an integral part of people's lives,  and for some people it might be difficult to actually live without them.  And I think when AR is likely to do the same, when it's focused on the things people actually need,  and once the technology comes along.  But before we go there, a short, very short history of augmented reality so that you can see where this is all coming from.  So actually augmented reality is rather old.  So the first augmented reality, virtual reality device, was built by Ivan Sutherland in 1968.  It was called sort of a thermoclass, but it had already had tracking and see-through optics.  So that's quite advanced for the time, I would say.  The actual term augmented reality was only coined 25 years later by Kauder and Mitzer in 1992.  So quite a long time in between, but of course the research still stood not during that period.  Here's an interesting example that I wanted to show you from 1994 by Stade et al.  which shows an ultrasonic image of a human fetus directly superimposed over a woman's belly.  And yet it shows that even very early on that people were thinking about how we can use augmented reality  to really benefit humans in a lot of ways, despite the technology at that time being very limited.  This is also an important step because it was the first mobile AR headset for usage outside of the lab  with a wonderful name Turing Machine by Fine Art Art in 1997.  So even like early on the question was okay, how can we use this technology not only indoors,  but how can we use it outdoors as well, which is very important.  And then finally the first AR application using mobile devices, in this case PDAs for the people who still remember them,  with extended cameras.  And you can see those little markers in the background which are tracked by the cameras  and then little virtual trains are overlaid over the wooden train tracks.  So it's not really that practical just to like a proof of concept more or less.  So but yeah, like I said mobile AR today is the most common form of augmented reality,  but maybe that might change soon because now we have like the currently the first generation of really  commercially available augmented reality headsets and therefore also an increased interest  from the scientific community, but also from the industry for solving their various problems.  So this immediately brings us to the areas of application and hopefully how augmented reality  may influence our lives in a positive way.  First a little bit about the research.  Actually only 12.5 percent of the research is actually focused on application cases for augmented reality  based, the statistics are based on the international symposium on augmented reality.  And the rest is focused on things like tracking technology, interaction, rendering and so on.  And of those application cases most of them are usually industry centered and not so much  on everyday life.  So we have things like for example logistics, you have things in the medical sectors and of course  they have a great indirect influence on our lives, we will still benefit from those things,  but I want to focus a little bit more on everyday use, so how can we immediately benefit from  augmented reality and explore that a little further.  And those are things like guides and cultural heritage for example and tailor communication  and tailor presence, which also brings us to the first use case which is AR tailor presence,  which is more or less like video phoning only more with 3D avatars of the person you're speaking  with embedded directly into the environment.  So basically like a virtual avatar sitting on the chair next to you, which is I would say  rather cool because it's something that's out of sci-fi novels or films, but now it's here.  Now we can use it and despite being far from perfect, it's still very incredible,  at least from my perspective.  And you can not only have a conversation in real size, but you can also scale it up and down  as much as you want, like in the lower right image you can see that the girl is standing on  the pedestal in front of the guy, very small in size, you can actually have people,  virtual people running around your table where you do other stuff while talking to them.  So another very classic example I would say is AR navigation.  This is a system by Google, which is a direct advancement over their smartphone navigation.  And I think despite being a very obvious example of when you think what AR can do for you,  it's also very powerful one.  So basically it means like when you want to go somewhere, you project like for example a line  in front of the people that you have to follow to reach the destination.  It's very easy and very understandable.  Currently there are mostly specialized systems that focus on something like airport navigation  or car navigation for example, very few for general pedestrian use, but I think that's likely  going to change in the future.  So the next one also maybe a little obvious finding places of interest based on selected  preferences.  So the example would be like where's the next post box, where's a good restaurant or bar.  So this is of course possible today with smartphones as well, but the big difference is  that you don't have any abstraction.  So you don't have to match the map with what you're seeing.  So where is this building that it says on my smartphone.  You get the information placed exactly at the point of interest itself.  So that makes it a lot of easier and useful for example.  Of course you can also show something like user reviews or for example the menu of the restaurant  so that you can find a suitable match.  What's also very, very useful was something like AR translation, where you like provide an  instant translation of a foreign language in place.  So for example, you look at a sign in Russian and you get the content in English.  This is actually Google translate, which maybe some people already have used.  Yeah, what's like you would when you have used it, you would say, oh, well, that's really practical  for example.  And when you use it with a head mounted device, it's even more practical because you don't  actually have to point a device somewhere to just see the text and the language you prefer.  So but besides translation, you can also offer context information as well of course,  because you can not only show in this example the direction, but you can also show the distance to  the object.  And it's also suitable for places like museums, where space for text is usually very limited,  but space in AR generally is not.  Which brings us to the next application case, which is cultural heritage and tourism.  And this is a nice paper that gives a good example of that because it can make history  somewhat more experienceable.  So when you have ruins, for example, of a Greek temple like here, you can get an impression  of what the temple might have looked like directly situated on the actual ruins.  Or when you have the fields of antique running tracks, you may have a virtual efflates on  augmented reality that give you an impression how these facilities were actually used, like  the people running around doing sports and stuff like that.  And I think this is really powerful.  I think you just get to see what it was looked like in the past.  But another important aspect, and I think a very powerful one, is how AR may have the  power to make technology more accessible again.  So today's technology grows more complex and it's more and more connected.  So that's even for experts, it's sometimes hard to actually say, okay, what state is the system  currently in?  For a layman person, it's even harder.  So AR can help make this hidden information visible again in a way that's easy to understand.  So just to give you a little example, which is not perfect, but hopefully sufficient to transport  the idea, imagine you have a simple router at home, for example.  You have some status LEDs, but for whatever reason, your internet breaks down, stops working,  they don't have you that much for debugging the actual problem.  So to get more information, you have to most likely access a web interface if you still can  remember the IP of your device.  Of course, you can do what we have done here and put a little piece of paper with the IP  on the device itself, but that's hardly a practical solution.  Yeah, so imagine with augmented reality, you can just look at the router, do a simple gesture,  and then get the information you were interested in directly superimposed on the physical device  itself, which can help you immediately recognize what the problem might be.  So maybe you just forgot to turn on DHCP or something like that.  But we can expand that concept to not only include a router, but also the wireless devices  connected to it.  So one of the predominant features of wireless LAN is that there isn't a wire,  so it's hard to see which devices are connected, where they are located, and what the IP is, and so on.  So where we are, we can simply draw virtual connections to make that information easily  available to the user.  So I do something like this.  But we can also use it to make errors more recognizable.  For example, one of the devices may be not be able to resolve websites because there is no  DNS server registered.  And then we can, of course, also display additional information like the bandwidth  the device uses, and when we have an actual physical medium, we could actually use that for  displaying information like a graph of the bandwidth of a time.  And it's very easy to understand what is displayed because it's directly located at the medium itself.  Yeah, so I hope this example gives you an impression of how AR can really help  with understanding technology again.  We might talk a little bit about that later on.  But first, let's come to some technical challenges.  So how does that actually work, and what has to change so that we can really use it every day?  The first one is tracking, which is one of the main challenges, because I have to locate the  user somehow.  I have to know where he or she is and what she's looking at, for example.  So for general positioning, I could use something like GPS, but precision is, of course, a  problem, and I can't really use it indoors.  So we need additional tracking technology.  And there are many different approaches, but the current state of the art is more or less  infrared tracking with time of flight.  So that means infrared light is emitted, which is reflected by the environment, and the  time between sending and receiving the light impulse determines the distance to an object.  And with that, I can create a depth model of a room, which more or less looks like this.  And then I can not only tell where the user is right now, but also when he or she is  moving around.  And while this technology is far from perfect, I think the current tracking technology is  quite robust and useful in that regard.  What's also important is display and rendering.  So there are many different technologies for head mounted devices.  One of them is video see-through, where camera films the environment, which is then together  with the virtual content rendered onto a display in front of the USSR.  But latency is a very big problem in that regard, because you will very easily get sick.  Another is see-through optics using polarized glass, where the air content is projected onto.  And there are even small laser projectors, which project directly into the user's eyes.  But for some reason, user acceptance is not very high.  But the ideal display would not require any external display at all, but you would directly  manipulate the optic nerve.  Which sends the messages to the brain.  Well, we don't have that right now, so we have to make through what we have.  And this is basically see-through optics, which is right now like the current state of the art.  So let's look at what the problems are in that regard.  One of the problems is opacity, because when I have a solid object, I have a solid object.  Because when I have a solid object, like a human for example, no background should bleed through them.  So currently for the commercial headsets, this is not so bad.  But really depending on the lighting conditions, when you have a lot of sunlight,  you can't really see anything anymore.  But yeah, there's still much room for improvement in that regard.  What might be even more important is the field of view of the AR headset,  because like a human has roughly 210 degrees field of view.  And as you can see here, like the Microsoft HoloLens has like 30 degrees,  and the Meta 2 has like 90 degrees, most of which for my experience at least are pretty blurry.  So yeah, that's not...  There still has to get a lot better until we can really use it.  Also very important,  when a person wearing an HMD, what does she see?  Like how can I recognize what a person is looking at?  And how can I get data on the recognized objects?  So due to performance requirements, most of the processing will most likely not happen  on the head mounted device, but it will very likely take the form of some web service or similar services.  Which not only process a constant visual input stream of the cameras of the HMD,  but also have to provide data sources for the various objects.  For example, when I have touristic points of interest, I need to get some information on them.  So this requires a constant data stream and a very good connectivity to make it really useful.  And it also requires a lot of services readily available to use by the AR device.  So this is quite a big challenge, maybe the biggest of them all.  So to give you a little impression how that could work based on existing research,  which is called a reality-based information retrieval.  So imagine you have an AR HMD that feeds a constant live image to a server to recognize objects  the user is currently looking at, for example, vegetables at a market.  So the server is able to classify those images by comparing them with a couple of saved reference images  and gives a list of recognized objects and the confidence value back to the HMD.  And then the HMD can query a different servers to get some recommendations for meals  that I can cook with that vegetable.  So when I explore a market and you have your AR display with you,  you could not only tell you what a particular vegetable is in case you might not know,  but it could also tell you what to do with it, which more or less could be quite useful.  What's also important, like, if AR glasses are to become everyday objects,  economics and aesthetics are going to be important.  So you probably don't want to wear something like this right now.  So for the most part, that means shrinking the glasses down to the size of current normal glasses,  which was a form factor that's more or less already accepted in our society.  And as you can see, in the comparison, we are not really close to being at that point.  And one of the most important questions is not only how to display the AR content,  but also how to interact with the content, because otherwise it's a very limited experience.  So our AR objects might appear to be part of our environment.  We can't really touch them.  So I can't actually press a virtual button.  And there is research concerned with providing haptics as well.  For example, there is some ultra haptics, which uses ultrasound to create resistance in midair,  so that I have an actual resistance when I touch somewhere, but it's still in a very early state.  So currently, mostly gesture interfaces are used, where my fingers are tracked,  and the line of sight together with the hand gestures are used to trigger interaction.  Speech input is also a common solution, but use in public environments is limited.  And another possibility is also to use additional devices like specialized controllers,  or for example, the touchscreen of a smartphone for interacting with the virtual content.  Gaze tracking, that means tracking the user's eyes, will most likely also play an important role.  But right now, there is pretty much no perfect solution, and also still a lot of research happens in this regard.  So this is a very hot research topic right now.  So let's come to possible dangers and other issues.  So all the things that might not be so great about ubiquitous augmented reality.  The first one, pretty obvious problem, is information overload,  where too much information leads to excessive cognitive load by users.  So virtual content has to be used sparsely to best support the users,  and the only way to make sure users are not overstrained is to leave them in control of what and how much is displayed.  But users might not always have the means to control it.  So especially when third party services are unwarfed, we may see something like  invasive advertisement, sorry, and other unwanted content.  And since head-mounted AR is already a personal experience,  these ads are most likely to be personalized as well.  So this directly leads to the question of privacy and data security.  There will most likely be a constant stream of the user's current position and a live video feed to various third party services.  So this raises the old question, how can I control what happens with my data and who gets access to it?  And privacy, of course, is also a big issue, because there's hardly anything more personal than a constant live feed of what I'm seeing.  So even if I myself decide to don't use an AR device, as long as everyone else does, it hardly makes a difference.  So at the center maybe is really the question of who controls the actual hard on software platforms.  So similar to smartphones, it's likely for a few global operating corporations to not only offer the AR devices,  but also provide the corresponding software ecosystems as well.  So we see that already with the direction the HoloLens by Microsoft is taking.  But AR consists of even more sensitive data than current smartphones.  So will we be in control of our own devices?  Do the companies producing them have any interest to keep us in control?  Because most likely their primary motivation is to make money.  And will people even care?  Because they hardly seem to care with smartphones right now.  So I think this is an important question.  And another possible application area for augmented reality, which I haven't talked about, is military use.  So heads up displays have been used in the military in aeroplanes, for example, for over a decade.  And there also have been several scientific papers evaluating the use of augmented reality in military use.  So it's not difficult to imagine AR systems like telling soldiers where to go and maybe even who's friend or foe.  Microsoft has recently made a contract with the US military for over 480 million US dollars  to evaluate the use of the HoloLens on the battlefield and the goal is to increase  lethality, mobility and situational awareness of soldiers on the battlefield.  So I don't really want to provide any judgment on that, but it's best to not have any illusions  what this technology can and eventually will be used for.  And it's probably not all going to be unicorns and rainbows.  So let's keep that in mind.  Yeah, another interesting aspect is how does massive use of augmented reality actually change  how humans interact with each other?  So for example, when we can project ourselves everywhere at any time, how important will it be to meet in person?  So to give you an impression, there's the novel Naked Sun written by Isaac Asimov in 1957,  where it's common for people to communicate with photorealistic holograms and even being  naked in front of each other.  There's no problem, but it's extremely unpleasant for people to be in the same room with a real person.  So I'm not going to say this is what is likely to happen, but I think it's an interesting aspect  to think about as there will be some social changes involved, which we don't know yet.  This also leads to the question how our own way of thinking is influenced by the constant use of augmented reality.  So for example, how does our perception of locations and how locations are connected  change when we are guided by AR navigation, which in the worst case means following a line until we reach our destination?  There has been research showing that extensive usage of GPS might lead to an atrophy of the hippocampus,  increasing the risks of cognitive diseases like Alzheimer's later on in life.  So it's important to be aware that there might be issues and it's important to do the actual research to figure out what these issues might be.  And of course, maybe even the ultimate question, what is reality?  Like how does our understanding and perception of reality itself change when we use AR?  This is a metaphysical question, but I think it's an interesting one nonetheless.  It's not so much a problem for moderate use of augmented reality, for example, to annotate objects,  but more a question for photo realistic augmented reality, which can alter the way the world looks to each person individually.  So if my world looks vastly different from your world and when AR is always active, what does reality really mean in this regard?  And what role, if any, does the unaugmented world play in the lives of people anymore?  So this might be something to think about.  And with that, I conclude my talk.  So hopefully I could show you that augmented reality could have a benefit in our life,  but that there are also some risks involved and I hope you found it somewhat interesting.  And now we can have some questions. Thank you.  Thank you for your fantastic talk, good overview of whatever is possible.  I see some people here queuing to ask you some questions.  Let's go to number two here. Please, sir.  Thanks for the talk. The previous slide on the red and blue pill.  Can you elaborate a little bit on how you as a researcher and user of augmented reality eventually better understand reality?  So you mean when we use augmented reality that might lead to a little better understanding of what reality is or what did you mean?  Yes, that's the question. Do you feel like you're not just using but also researching because in order to, for example,  when you have this slide on how the 3D scan works, you use it as, let's say, a metaphor or a shortcut on how our own brain and visual system works?  Yeah. Well, like I said, it's a rather metaphysical question.  So it's hard to really answer the question right now.  It's more like you could guess what it might, how it might change for people, what the perception of reality, how does that change?  But I don't, I can't really answer that, to be honest.  Like it's more of an open question for myself as well because like when I think about augmented reality and if it eventually will be used by everybody every day,  like I can think of some like dystopian sci-fi scenarios where everyone lives in this happy world, which is totally fake.  And the real world basically looks awful, but nobody actually noticed that because everyone is like caged in his own happy place, for example.  But I don't know if that's going to happen. It's just a possible danger that you could think of.  But this is probably something that will not happen in the next 20 to 30 years because we simply don't have the technology to do that right now.  So maybe that more or less answers your question, I hope.  Yeah, next one.  Yeah, I was thinking along as well.  Okay.  Sorry.  So to get widespread consumer adoption, we really need the miniaturization of the hardware.  Google Glass got kind of laughed out of the room a few years ago, so we need to be better than that.  Yeah, but Google Glass wasn't really augmented reality.  It was more like a headset display because you had this tiny screen in the back of the upper part of your field of vision.  And it was like you could just display a limited amount of information there.  Like when you have real augmented reality, it's basically having the whole space to put information.  Sure. I mean more like it's a stupid looking thing to walk around on your head.  So we've got to look better than that when we're for the mass consumer adoption.  But that's why I think that we have to go to where medical glasses are right now,  because they are already accepted in society.  So someone wearing glasses doesn't get looked at in a funny way because it's pretty normal.  So if augmented reality headsets look more or less the same, I think we're fine.  So the question is what are the technological problems that we have to solve to get there in a consumer device?  Yeah, we have to actually miniaturize the stuff.  So I think the laser projectors actually are an interesting solution because they are rather small  and you can project directly into the eyes of the users.  So if we get this user bias figured out, how we can actually convince people that this is a good idea, not dangerous at all,  we can shrink it down even further.  And if we have the whole processing, not on the device itself, but on third party, like data centers or something like that,  we can also shrink that down so we have to actually only stream the input to the server  and we have to stream the image back to the device.  Now we can build pretty small devices, I think.  So that's what we have to figure out.  But yeah, I think this is something that's actually going to happen in the next five years or something like that,  because this is actually something that's worked on right now.  Microsoft actually is building the next version of HoloLens,  which should be smaller, which should have more features like eye tracking and stuff like that.  But we have to wait and see, basically.  This is also an important thing.  The hardware involved is very complex.  So it's not something that one person can do in his garage or something like that.  So you pretty much only have the chance to have those big corporations do this stuff and do the research because it's so complex.  With a little bit of imagination, we see someone stepping in there from the Internet who has a question.  Olographically projected here next to me on stage.  The Internet has a related question.  How long do you think will it take until everyone uses augmented reality like mobile phones today?  I think that really depends on the use cases.  I think this is the most important thing for acceptance.  You have to provide functionality that actually benefits the way people live their lives.  For example, the R translation is a good example because it's common for people when they are travelling, for example,  to run into signs or something like that, which they can't understand because it's a foreign language.  And if you provide an augmented reality solution for this particular problem, people are going to use it because it's practical,  because it's useful.  And the more functionality, it's the same with smartphones, really.  The more functionality you provide that goes in this direction, the easier the adoption will be.  So yeah, that's basically, we have to find use cases which are important to people and then the adoption will basically happen itself.  Critical mouse, probably.  Yeah, critical mouse.  Number three, please, sir.  Thank you for your talk.  When you introduce the platforms, what is with Facebook with the Oculus Rift?  Is it not virtual reality?  Yeah, it's virtual reality.  It's not really augmented reality.  Yeah, I was thinking about including it, but I still don't have any idea where they are going with this.  So they have bought this company.  They have invested in this technology.  But yeah, I don't see where they are going with this right now, what their goal is actually.  So this is why I didn't include it.  Thank you.  Yeah, it will come in your direction anyhow.  But number two, please.  First of all, thank you for the informative talk.  And what do you think will change from privacy policy with when you have an active life feed of what you're seeing with augmented reality?  Yeah, I think like this may be the cynical way of looking at it,  but I think people will stop caring about that.  That is what simply is going to happen.  Like the current generation is concerned with privacy,  and we feel like for us it's a very bad idea to actually have a constant life feed of what we are seeing transmitted to somewhere on the internet.  But I see that with younger people who are smart phones.  They don't think about privacy pretty much.  They think about what this device can do for them.  And I think this is like more or less the social change.  Maybe that will happen.  People will stop caring about it.  It will just be a normal thing that your field of vision gets transmitted to somewhere on the internet,  and people will simply stop care.  I think that is what's going to happen here.  Thank you.  Okay, next number two.  Thanks.  I saw recently there was a company that announced some shades,  and they had bone conducting sound transmission.  So they didn't really project something into the retina,  so they went away around all the projection and all the complexity of making this huge bulky headset.  Do you think this is a viable way to go forward?  You've focused mostly on visual inputs,  but maybe other sensors are better suited to making an entry into augmented reality.  What do you think about that?  An interesting point.  I focus a lot on the visual side of things,  but the important thing, I guess, with all this mixed reality and virtual reality stuff,  is that it's not limited to the visual.  So basically it should be a whole sensory experience,  so all your senses are involved.  So yeah, maybe that's a good idea to not only focus on the visuals so much,  but also maybe on things like audio, like you said.  It's really the same question with the use cases.  When you figure out a very good use case where an audio augmentation helps people in their everyday life,  I think people will start to use it.  But the reason why I focus mostly on the visual part is because I think it's the most strongest sensory experience people usually have,  one day have eyesight at least.  And I think this is most likely also to be like the selling point of augmented reality,  where most likely to be the visual part.  Okay, another one.  Number two.  Thanks for the talk.  You mentioned that one of the ways you see the miniaturization of these devices evolving is by streaming data into the cloud.  But we've seen, I mean, especially with smartphones these days having the equivalent computational power of computers from a decade ago,  that it is actually possible to do a lot of these machine vision tasks on a smartphone.  And I think some manufacturers are basically trying to get your smartphone to be your primary computer.  So do you see that as a likely alternative?  Because obviously this comes back to the privacy question because if you can keep things local.  Yeah, but it also comes back to what the producers of the devices want you to actually do with the devices.  Yeah, of course, when you have miniaturization it is powerful enough to put a whole computer into something like normal glasses.  You could do that and you don't have to stream that much information.  But I guess what you still have to actually do all this stuff because you are not alone in the world basically with augmented reality.  So if everyone else is using augmented reality you need to have that actual data from what the others are doing.  Like you have to build, for example, in this building, I have to build a constant room model of how this building looks like.  So everyone who is moving through this building will constantly scan the environment to build the 3D model of the environment.  But you have to share that information so that when I move into this building for the first time I still have the whole environment mapped out  because someone else already walked through it.  So you still have to stream all that and you still have to share all those data.  So I think even if you increase the processing power of the devices itself, this is not going to change much.  Okay, sir.  Okay, my question is very much related.  With all the big players designing the devices, it's still a closed world.  And I would have to imagine I would have to stream my live feed to some Microsoft server or whatever.  Are there any efforts to keep at least interfaces open if not the whole software open source?  So for example, the CCC could put up a server I trust and send my live feed to to get a process.  The system is open or not that you can actually change the location where your information is transmitted to.  Yeah, well, at least I don't know of any open hardware, open software projects who try to do augmented reality right now.  Of course, there's research.  Research is more or less open most of the time, but research prototypes right now, while they are good at what they do, they are hardly like consumer products.  So you can't really mass produce them and distribute them to the people.  So yeah, you probably could do it, but I don't see currently I don't see any movement in this regard.  Yeah.  I shouldn't do that, but just for the previous question, I would suggest.  Speak up, please.  I shouldn't do that, but to answer briefly the previous question, WebAR and some effort from Mozilla are trying to bring at least the highest level of the stack in an open source fashion for augmented reality.  Okay.  Yeah, I was wondering for the example you showed on your research in your own lab about information retrieval.  Those examples that are able to distinguish between one object and another, let's say an apple and a pear or a tomato works well, but I'm wondering how scalable it is because when you get to a morphicized example.  Yeah, that's a good point because like what we were using was an actual food database for that.  So it was very easy to classify like the vegetables because they are food.  But if you have a database with everything in it, of course, it's a problem because like classification will get you some weird results.  So maybe the vegetable would be recognized.  I don't know as a lamp or something like that.  Yeah, that's that's a problem.  But this is only a problem like in this particular research case, it was more like a proof of concept to actually show. Okay, how can we use augmented reality in some more practical everyday scenarios.  It wasn't really much focused on to find the perfect technical solution for providing like the classification of the images.  This is also why we use an existing service for that and we didn't build our own.  So we just use what was available.  Okay, unless someone is beaming himself up from the Internet, then we had all the questions here answered.  Thank you very, very much.  Thank you.  I hope to see you soon.
35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_0_1259.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_1260_1391.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_1392_1895.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_1896_20363.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_20364_22019.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_22020_25115.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_25116_25643.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_25644_25931.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_25932_30443.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_30444_42215.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_42216_42983.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_42984_45323.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_45324_46799.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_46800_50615.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_50616_51647.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_51648_51755.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_51756_52403.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_52404_54827.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_54828_55343.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_55344_55547.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_55548_56843.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_56844_57515.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_57516_57995.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_57996_58319.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_58320_60311.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_60312_60371.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_60372_60707.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_60708_60995.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_60996_61211.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_61212_61463.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_61464_61511.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_61512_62039.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_62040_64799.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_64800_67103.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_67104_67211.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_67212_68051.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_68052_68255.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_68256_68303.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_68304_69299.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_69300_70799.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_70800_71039.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_71040_72155.png 35c3-9623-augmented_reality_bridging_the_gap_between_the_physical_and_the_digital_world_72156_72968.png
We're going to look at how to bridge the gap between the physical and the digital world. So we go into immersive things. And my guest here, our guest here is Patrick, and he goes as well with the nickname, Prype. He's currently doing a PhD on human-computer relations interaction. I'm not going to discuss this since I'm a little bit on this field. We made an agreement about that. But he's focusing on augmented reality today. He's doing this study in Dresden, I understood. He has a general interest of course in things to do with 3D. That goes along with these kind of things. And his talk provides us an overview on AR in general and explains its possible uses for goods and for the bad. Ladies and gentlemen, here is Patrick. Please give him a warm welcome. Thank you. Like the views and put it in play, Patrick. Thanks. Hi everyone. This is a foundation talk. It's mostly aimed at people who know nothing about augmented reality. So, hopefully after the talk you will know a little bit more. However, if you have some experience, I hope maybe you will get some new impulses. I don't know. Furthermore, the talk is also focused more about how augmented reality can be used for everyday life and not so much on the scientific community. And it's roughly 30 minutes or something like that. So, I made a little shorter so that we have some questions in the end. So, if you have something that's of particular interest, just ask afterwards. So, but first let's talk a little bit about virtual reality because I think most people maybe are more familiar with the term. There has been a lot of talk about virtual reality like changing and revolutionizing the way we use computers. There are as many commercial virtual reality assets as ever before and also quite a lot of applications. But somehow we are doesn't really influence our everyday lives that much. And I think there's two reasons for that. The first one is that virtual reality applications are mostly centered on entertainment right now. And the second is that it disconnects us pretty much from the real world. So, why it might have the ability to actually transfer us into another world which more or less looks like this. At the same time we are cut up from reality and in contrast augmented reality expands the real world. And in the mind of most people right now this looks something like that. But the important thing is that we can still perceive our surroundings and we can also experience the virtual content as well. And this particular difference makes AR a lot more useful for everyday use than virtual reality. So let's go to a little definition of what augmented reality actually means. So the important thing is that we combine real and virtual content, blowing the borders between both. So as you can see here's a virtual globe on a real table. And the application also interacts in real time. So if I move or if I have some input, the application reacts on the spot. There's no delay. And the objects are situated not relative to the user but in space. So if I move around the globe stays on the table. So this is basically what I'm talking about augmented reality, what I mean. But what is also important to know that AR is only part of the so-called mixed reality continuum, which ranges from reality without any augmentations. So the normal reality we experience every day to a completely virtual environment. And those four categories are basically the classic mixed reality continuum. There's also some extensions to that, which you can see here. I don't want to go into any detail of what those all mean. But the important thing to remember is that there is no hard border where augmented reality begins and where it ends. There's also a lot of ways to provide augmented reality. Mobile augmented reality like Pokemon Go, where the AR content is perceived to a mobile device like a smartphone, is the most common form today. There's also projected augmented reality, where you have a projector superimpose an image onto a real environment, which is especially useful if you have like collaborative scenarios where the content should be perceived and manipulated by a group of people at the same time. You can actually see this example right out of the door. There's a similar installation there. And then there's what this talk is also mostly focused about, head-mounted augmented reality, where users wear AR glasses, which can be seen through, and where the virtual content is projected onto. And this is the most immersive form of augmented reality. And when you have sophisticated technology, it's more or less indistinguishable from the real world. So that's basically the goal where we want to get. So, but now it's the question, OK, why augmented reality? What's so important? Why do I have to talk at all? And yeah, for that, I want to quote Tim Sweeney of the CEO of Epic Games, because it's always a good idea to quote someone who's more famous than oneself. And he said that he believes that augmented reality will be the biggest technological revolution in our lifetime, because once you have an AR display, you don't need any other form of display. You just take your screen wherever you go. So that's a very powerful thing to say. You also think that the revolution will happen in the next 10 years, which is very optimistic from my perspective when you look at what the current technology can do. But nonetheless, I believe this revolution is going to happen sooner or later. So it's useful to familiarize yourself with the term. What is also important to note is that the goal is not to just substitute a real display with virtual ones, but augmented physical objects themselves, that the augmentations really have a relation with the location they are situated in. So only then AR will have really the power to revolutionize our everyday lives. So I will elaborate on that a little later on. But first, a question like how likely are fundamental changes in our society triggered by augmented reality? So if we compare it with smartphones and the way they are used, I think it's very likely. Because smartphones today are technology focused directly on supporting the way people live their lives every day. So it also started as a research a long time ago, before commercial products were available, and now they are ubiquitous, they are an integral part of people's lives, and for some people it might be difficult to actually live without them. And I think when AR is likely to do the same, when it's focused on the things people actually need, and once the technology comes along. But before we go there, a short, very short history of augmented reality so that you can see where this is all coming from. So actually augmented reality is rather old. So the first augmented reality, virtual reality device, was built by Ivan Sutherland in 1968. It was called sort of a thermoclass, but it had already had tracking and see-through optics. So that's quite advanced for the time, I would say. The actual term augmented reality was only coined 25 years later by Kauder and Mitzer in 1992. So quite a long time in between, but of course the research still stood not during that period. Here's an interesting example that I wanted to show you from 1994 by Stade et al. which shows an ultrasonic image of a human fetus directly superimposed over a woman's belly. And yet it shows that even very early on that people were thinking about how we can use augmented reality to really benefit humans in a lot of ways, despite the technology at that time being very limited. This is also an important step because it was the first mobile AR headset for usage outside of the lab with a wonderful name Turing Machine by Fine Art Art in 1997. So even like early on the question was okay, how can we use this technology not only indoors, but how can we use it outdoors as well, which is very important. And then finally the first AR application using mobile devices, in this case PDAs for the people who still remember them, with extended cameras. And you can see those little markers in the background which are tracked by the cameras and then little virtual trains are overlaid over the wooden train tracks. So it's not really that practical just to like a proof of concept more or less. So but yeah, like I said mobile AR today is the most common form of augmented reality, but maybe that might change soon because now we have like the currently the first generation of really commercially available augmented reality headsets and therefore also an increased interest from the scientific community, but also from the industry for solving their various problems. So this immediately brings us to the areas of application and hopefully how augmented reality may influence our lives in a positive way. First a little bit about the research. Actually only 12.5 percent of the research is actually focused on application cases for augmented reality based, the statistics are based on the international symposium on augmented reality. And the rest is focused on things like tracking technology, interaction, rendering and so on. And of those application cases most of them are usually industry centered and not so much on everyday life. So we have things like for example logistics, you have things in the medical sectors and of course they have a great indirect influence on our lives, we will still benefit from those things, but I want to focus a little bit more on everyday use, so how can we immediately benefit from augmented reality and explore that a little further. And those are things like guides and cultural heritage for example and tailor communication and tailor presence, which also brings us to the first use case which is AR tailor presence, which is more or less like video phoning only more with 3D avatars of the person you're speaking with embedded directly into the environment. So basically like a virtual avatar sitting on the chair next to you, which is I would say rather cool because it's something that's out of sci-fi novels or films, but now it's here. Now we can use it and despite being far from perfect, it's still very incredible, at least from my perspective. And you can not only have a conversation in real size, but you can also scale it up and down as much as you want, like in the lower right image you can see that the girl is standing on the pedestal in front of the guy, very small in size, you can actually have people, virtual people running around your table where you do other stuff while talking to them. So another very classic example I would say is AR navigation. This is a system by Google, which is a direct advancement over their smartphone navigation. And I think despite being a very obvious example of when you think what AR can do for you, it's also very powerful one. So basically it means like when you want to go somewhere, you project like for example a line in front of the people that you have to follow to reach the destination. It's very easy and very understandable. Currently there are mostly specialized systems that focus on something like airport navigation or car navigation for example, very few for general pedestrian use, but I think that's likely going to change in the future. So the next one also maybe a little obvious finding places of interest based on selected preferences. So the example would be like where's the next post box, where's a good restaurant or bar. So this is of course possible today with smartphones as well, but the big difference is that you don't have any abstraction. So you don't have to match the map with what you're seeing. So where is this building that it says on my smartphone. You get the information placed exactly at the point of interest itself. So that makes it a lot of easier and useful for example. Of course you can also show something like user reviews or for example the menu of the restaurant so that you can find a suitable match. What's also very, very useful was something like AR translation, where you like provide an instant translation of a foreign language in place. So for example, you look at a sign in Russian and you get the content in English. This is actually Google translate, which maybe some people already have used. Yeah, what's like you would when you have used it, you would say, oh, well, that's really practical for example. And when you use it with a head mounted device, it's even more practical because you don't actually have to point a device somewhere to just see the text and the language you prefer. So but besides translation, you can also offer context information as well of course, because you can not only show in this example the direction, but you can also show the distance to the object. And it's also suitable for places like museums, where space for text is usually very limited, but space in AR generally is not. Which brings us to the next application case, which is cultural heritage and tourism. And this is a nice paper that gives a good example of that because it can make history somewhat more experienceable. So when you have ruins, for example, of a Greek temple like here, you can get an impression of what the temple might have looked like directly situated on the actual ruins. Or when you have the fields of antique running tracks, you may have a virtual efflates on augmented reality that give you an impression how these facilities were actually used, like the people running around doing sports and stuff like that. And I think this is really powerful. I think you just get to see what it was looked like in the past. But another important aspect, and I think a very powerful one, is how AR may have the power to make technology more accessible again. So today's technology grows more complex and it's more and more connected. So that's even for experts, it's sometimes hard to actually say, okay, what state is the system currently in? For a layman person, it's even harder. So AR can help make this hidden information visible again in a way that's easy to understand. So just to give you a little example, which is not perfect, but hopefully sufficient to transport the idea, imagine you have a simple router at home, for example. You have some status LEDs, but for whatever reason, your internet breaks down, stops working, they don't have you that much for debugging the actual problem. So to get more information, you have to most likely access a web interface if you still can remember the IP of your device. Of course, you can do what we have done here and put a little piece of paper with the IP on the device itself, but that's hardly a practical solution. Yeah, so imagine with augmented reality, you can just look at the router, do a simple gesture, and then get the information you were interested in directly superimposed on the physical device itself, which can help you immediately recognize what the problem might be. So maybe you just forgot to turn on DHCP or something like that. But we can expand that concept to not only include a router, but also the wireless devices connected to it. So one of the predominant features of wireless LAN is that there isn't a wire, so it's hard to see which devices are connected, where they are located, and what the IP is, and so on. So where we are, we can simply draw virtual connections to make that information easily available to the user. So I do something like this. But we can also use it to make errors more recognizable. For example, one of the devices may be not be able to resolve websites because there is no DNS server registered. And then we can, of course, also display additional information like the bandwidth the device uses, and when we have an actual physical medium, we could actually use that for displaying information like a graph of the bandwidth of a time. And it's very easy to understand what is displayed because it's directly located at the medium itself. Yeah, so I hope this example gives you an impression of how AR can really help with understanding technology again. We might talk a little bit about that later on. But first, let's come to some technical challenges. So how does that actually work, and what has to change so that we can really use it every day? The first one is tracking, which is one of the main challenges, because I have to locate the user somehow. I have to know where he or she is and what she's looking at, for example. So for general positioning, I could use something like GPS, but precision is, of course, a problem, and I can't really use it indoors. So we need additional tracking technology. And there are many different approaches, but the current state of the art is more or less infrared tracking with time of flight. So that means infrared light is emitted, which is reflected by the environment, and the time between sending and receiving the light impulse determines the distance to an object. And with that, I can create a depth model of a room, which more or less looks like this. And then I can not only tell where the user is right now, but also when he or she is moving around. And while this technology is far from perfect, I think the current tracking technology is quite robust and useful in that regard. What's also important is display and rendering. So there are many different technologies for head mounted devices. One of them is video see-through, where camera films the environment, which is then together with the virtual content rendered onto a display in front of the USSR. But latency is a very big problem in that regard, because you will very easily get sick. Another is see-through optics using polarized glass, where the air content is projected onto. And there are even small laser projectors, which project directly into the user's eyes. But for some reason, user acceptance is not very high. But the ideal display would not require any external display at all, but you would directly manipulate the optic nerve. Which sends the messages to the brain. Well, we don't have that right now, so we have to make through what we have. And this is basically see-through optics, which is right now like the current state of the art. So let's look at what the problems are in that regard. One of the problems is opacity, because when I have a solid object, I have a solid object. Because when I have a solid object, like a human for example, no background should bleed through them. So currently for the commercial headsets, this is not so bad. But really depending on the lighting conditions, when you have a lot of sunlight, you can't really see anything anymore. But yeah, there's still much room for improvement in that regard. What might be even more important is the field of view of the AR headset, because like a human has roughly 210 degrees field of view. And as you can see here, like the Microsoft HoloLens has like 30 degrees, and the Meta 2 has like 90 degrees, most of which for my experience at least are pretty blurry. So yeah, that's not... There still has to get a lot better until we can really use it. Also very important, when a person wearing an HMD, what does she see? Like how can I recognize what a person is looking at? And how can I get data on the recognized objects? So due to performance requirements, most of the processing will most likely not happen on the head mounted device, but it will very likely take the form of some web service or similar services. Which not only process a constant visual input stream of the cameras of the HMD, but also have to provide data sources for the various objects. For example, when I have touristic points of interest, I need to get some information on them. So this requires a constant data stream and a very good connectivity to make it really useful. And it also requires a lot of services readily available to use by the AR device. So this is quite a big challenge, maybe the biggest of them all. So to give you a little impression how that could work based on existing research, which is called a reality-based information retrieval. So imagine you have an AR HMD that feeds a constant live image to a server to recognize objects the user is currently looking at, for example, vegetables at a market. So the server is able to classify those images by comparing them with a couple of saved reference images and gives a list of recognized objects and the confidence value back to the HMD. And then the HMD can query a different servers to get some recommendations for meals that I can cook with that vegetable. So when I explore a market and you have your AR display with you, you could not only tell you what a particular vegetable is in case you might not know, but it could also tell you what to do with it, which more or less could be quite useful. What's also important, like, if AR glasses are to become everyday objects, economics and aesthetics are going to be important. So you probably don't want to wear something like this right now. So for the most part, that means shrinking the glasses down to the size of current normal glasses, which was a form factor that's more or less already accepted in our society. And as you can see, in the comparison, we are not really close to being at that point. And one of the most important questions is not only how to display the AR content, but also how to interact with the content, because otherwise it's a very limited experience. So our AR objects might appear to be part of our environment. We can't really touch them. So I can't actually press a virtual button. And there is research concerned with providing haptics as well. For example, there is some ultra haptics, which uses ultrasound to create resistance in midair, so that I have an actual resistance when I touch somewhere, but it's still in a very early state. So currently, mostly gesture interfaces are used, where my fingers are tracked, and the line of sight together with the hand gestures are used to trigger interaction. Speech input is also a common solution, but use in public environments is limited. And another possibility is also to use additional devices like specialized controllers, or for example, the touchscreen of a smartphone for interacting with the virtual content. Gaze tracking, that means tracking the user's eyes, will most likely also play an important role. But right now, there is pretty much no perfect solution, and also still a lot of research happens in this regard. So this is a very hot research topic right now. So let's come to possible dangers and other issues. So all the things that might not be so great about ubiquitous augmented reality. The first one, pretty obvious problem, is information overload, where too much information leads to excessive cognitive load by users. So virtual content has to be used sparsely to best support the users, and the only way to make sure users are not overstrained is to leave them in control of what and how much is displayed. But users might not always have the means to control it. So especially when third party services are unwarfed, we may see something like invasive advertisement, sorry, and other unwanted content. And since head-mounted AR is already a personal experience, these ads are most likely to be personalized as well. So this directly leads to the question of privacy and data security. There will most likely be a constant stream of the user's current position and a live video feed to various third party services. So this raises the old question, how can I control what happens with my data and who gets access to it? And privacy, of course, is also a big issue, because there's hardly anything more personal than a constant live feed of what I'm seeing. So even if I myself decide to don't use an AR device, as long as everyone else does, it hardly makes a difference. So at the center maybe is really the question of who controls the actual hard on software platforms. So similar to smartphones, it's likely for a few global operating corporations to not only offer the AR devices, but also provide the corresponding software ecosystems as well. So we see that already with the direction the HoloLens by Microsoft is taking. But AR consists of even more sensitive data than current smartphones. So will we be in control of our own devices? Do the companies producing them have any interest to keep us in control? Because most likely their primary motivation is to make money. And will people even care? Because they hardly seem to care with smartphones right now. So I think this is an important question. And another possible application area for augmented reality, which I haven't talked about, is military use. So heads up displays have been used in the military in aeroplanes, for example, for over a decade. And there also have been several scientific papers evaluating the use of augmented reality in military use. So it's not difficult to imagine AR systems like telling soldiers where to go and maybe even who's friend or foe. Microsoft has recently made a contract with the US military for over 480 million US dollars to evaluate the use of the HoloLens on the battlefield and the goal is to increase lethality, mobility and situational awareness of soldiers on the battlefield. So I don't really want to provide any judgment on that, but it's best to not have any illusions what this technology can and eventually will be used for. And it's probably not all going to be unicorns and rainbows. So let's keep that in mind. Yeah, another interesting aspect is how does massive use of augmented reality actually change how humans interact with each other? So for example, when we can project ourselves everywhere at any time, how important will it be to meet in person? So to give you an impression, there's the novel Naked Sun written by Isaac Asimov in 1957, where it's common for people to communicate with photorealistic holograms and even being naked in front of each other. There's no problem, but it's extremely unpleasant for people to be in the same room with a real person. So I'm not going to say this is what is likely to happen, but I think it's an interesting aspect to think about as there will be some social changes involved, which we don't know yet. This also leads to the question how our own way of thinking is influenced by the constant use of augmented reality. So for example, how does our perception of locations and how locations are connected change when we are guided by AR navigation, which in the worst case means following a line until we reach our destination? There has been research showing that extensive usage of GPS might lead to an atrophy of the hippocampus, increasing the risks of cognitive diseases like Alzheimer's later on in life. So it's important to be aware that there might be issues and it's important to do the actual research to figure out what these issues might be. And of course, maybe even the ultimate question, what is reality? Like how does our understanding and perception of reality itself change when we use AR? This is a metaphysical question, but I think it's an interesting one nonetheless. It's not so much a problem for moderate use of augmented reality, for example, to annotate objects, but more a question for photo realistic augmented reality, which can alter the way the world looks to each person individually. So if my world looks vastly different from your world and when AR is always active, what does reality really mean in this regard? And what role, if any, does the unaugmented world play in the lives of people anymore? So this might be something to think about. And with that, I conclude my talk. So hopefully I could show you that augmented reality could have a benefit in our life, but that there are also some risks involved and I hope you found it somewhat interesting. And now we can have some questions. Thank you. Thank you for your fantastic talk, good overview of whatever is possible. I see some people here queuing to ask you some questions. Let's go to number two here. Please, sir. Thanks for the talk. The previous slide on the red and blue pill. Can you elaborate a little bit on how you as a researcher and user of augmented reality eventually better understand reality? So you mean when we use augmented reality that might lead to a little better understanding of what reality is or what did you mean? Yes, that's the question. Do you feel like you're not just using but also researching because in order to, for example, when you have this slide on how the 3D scan works, you use it as, let's say, a metaphor or a shortcut on how our own brain and visual system works? Yeah. Well, like I said, it's a rather metaphysical question. So it's hard to really answer the question right now. It's more like you could guess what it might, how it might change for people, what the perception of reality, how does that change? But I don't, I can't really answer that, to be honest. Like it's more of an open question for myself as well because like when I think about augmented reality and if it eventually will be used by everybody every day, like I can think of some like dystopian sci-fi scenarios where everyone lives in this happy world, which is totally fake. And the real world basically looks awful, but nobody actually noticed that because everyone is like caged in his own happy place, for example. But I don't know if that's going to happen. It's just a possible danger that you could think of. But this is probably something that will not happen in the next 20 to 30 years because we simply don't have the technology to do that right now. So maybe that more or less answers your question, I hope. Yeah, next one. Yeah, I was thinking along as well. Okay. Sorry. So to get widespread consumer adoption, we really need the miniaturization of the hardware. Google Glass got kind of laughed out of the room a few years ago, so we need to be better than that. Yeah, but Google Glass wasn't really augmented reality. It was more like a headset display because you had this tiny screen in the back of the upper part of your field of vision. And it was like you could just display a limited amount of information there. Like when you have real augmented reality, it's basically having the whole space to put information. Sure. I mean more like it's a stupid looking thing to walk around on your head. So we've got to look better than that when we're for the mass consumer adoption. But that's why I think that we have to go to where medical glasses are right now, because they are already accepted in society. So someone wearing glasses doesn't get looked at in a funny way because it's pretty normal. So if augmented reality headsets look more or less the same, I think we're fine. So the question is what are the technological problems that we have to solve to get there in a consumer device? Yeah, we have to actually miniaturize the stuff. So I think the laser projectors actually are an interesting solution because they are rather small and you can project directly into the eyes of the users. So if we get this user bias figured out, how we can actually convince people that this is a good idea, not dangerous at all, we can shrink it down even further. And if we have the whole processing, not on the device itself, but on third party, like data centers or something like that, we can also shrink that down so we have to actually only stream the input to the server and we have to stream the image back to the device. Now we can build pretty small devices, I think. So that's what we have to figure out. But yeah, I think this is something that's actually going to happen in the next five years or something like that, because this is actually something that's worked on right now. Microsoft actually is building the next version of HoloLens, which should be smaller, which should have more features like eye tracking and stuff like that. But we have to wait and see, basically. This is also an important thing. The hardware involved is very complex. So it's not something that one person can do in his garage or something like that. So you pretty much only have the chance to have those big corporations do this stuff and do the research because it's so complex. With a little bit of imagination, we see someone stepping in there from the Internet who has a question. Olographically projected here next to me on stage. The Internet has a related question. How long do you think will it take until everyone uses augmented reality like mobile phones today? I think that really depends on the use cases. I think this is the most important thing for acceptance. You have to provide functionality that actually benefits the way people live their lives. For example, the R translation is a good example because it's common for people when they are travelling, for example, to run into signs or something like that, which they can't understand because it's a foreign language. And if you provide an augmented reality solution for this particular problem, people are going to use it because it's practical, because it's useful. And the more functionality, it's the same with smartphones, really. The more functionality you provide that goes in this direction, the easier the adoption will be. So yeah, that's basically, we have to find use cases which are important to people and then the adoption will basically happen itself. Critical mouse, probably. Yeah, critical mouse. Number three, please, sir. Thank you for your talk. When you introduce the platforms, what is with Facebook with the Oculus Rift? Is it not virtual reality? Yeah, it's virtual reality. It's not really augmented reality. Yeah, I was thinking about including it, but I still don't have any idea where they are going with this. So they have bought this company. They have invested in this technology. But yeah, I don't see where they are going with this right now, what their goal is actually. So this is why I didn't include it. Thank you. Yeah, it will come in your direction anyhow. But number two, please. First of all, thank you for the informative talk. And what do you think will change from privacy policy with when you have an active life feed of what you're seeing with augmented reality? Yeah, I think like this may be the cynical way of looking at it, but I think people will stop caring about that. That is what simply is going to happen. Like the current generation is concerned with privacy, and we feel like for us it's a very bad idea to actually have a constant life feed of what we are seeing transmitted to somewhere on the internet. But I see that with younger people who are smart phones. They don't think about privacy pretty much. They think about what this device can do for them. And I think this is like more or less the social change. Maybe that will happen. People will stop caring about it. It will just be a normal thing that your field of vision gets transmitted to somewhere on the internet, and people will simply stop care. I think that is what's going to happen here. Thank you. Okay, next number two. Thanks. I saw recently there was a company that announced some shades, and they had bone conducting sound transmission. So they didn't really project something into the retina, so they went away around all the projection and all the complexity of making this huge bulky headset. Do you think this is a viable way to go forward? You've focused mostly on visual inputs, but maybe other sensors are better suited to making an entry into augmented reality. What do you think about that? An interesting point. I focus a lot on the visual side of things, but the important thing, I guess, with all this mixed reality and virtual reality stuff, is that it's not limited to the visual. So basically it should be a whole sensory experience, so all your senses are involved. So yeah, maybe that's a good idea to not only focus on the visuals so much, but also maybe on things like audio, like you said. It's really the same question with the use cases. When you figure out a very good use case where an audio augmentation helps people in their everyday life, I think people will start to use it. But the reason why I focus mostly on the visual part is because I think it's the most strongest sensory experience people usually have, one day have eyesight at least. And I think this is most likely also to be like the selling point of augmented reality, where most likely to be the visual part. Okay, another one. Number two. Thanks for the talk. You mentioned that one of the ways you see the miniaturization of these devices evolving is by streaming data into the cloud. But we've seen, I mean, especially with smartphones these days having the equivalent computational power of computers from a decade ago, that it is actually possible to do a lot of these machine vision tasks on a smartphone. And I think some manufacturers are basically trying to get your smartphone to be your primary computer. So do you see that as a likely alternative? Because obviously this comes back to the privacy question because if you can keep things local. Yeah, but it also comes back to what the producers of the devices want you to actually do with the devices. Yeah, of course, when you have miniaturization it is powerful enough to put a whole computer into something like normal glasses. You could do that and you don't have to stream that much information. But I guess what you still have to actually do all this stuff because you are not alone in the world basically with augmented reality. So if everyone else is using augmented reality you need to have that actual data from what the others are doing. Like you have to build, for example, in this building, I have to build a constant room model of how this building looks like. So everyone who is moving through this building will constantly scan the environment to build the 3D model of the environment. But you have to share that information so that when I move into this building for the first time I still have the whole environment mapped out because someone else already walked through it. So you still have to stream all that and you still have to share all those data. So I think even if you increase the processing power of the devices itself, this is not going to change much. Okay, sir. Okay, my question is very much related. With all the big players designing the devices, it's still a closed world. And I would have to imagine I would have to stream my live feed to some Microsoft server or whatever. Are there any efforts to keep at least interfaces open if not the whole software open source? So for example, the CCC could put up a server I trust and send my live feed to to get a process. The system is open or not that you can actually change the location where your information is transmitted to. Yeah, well, at least I don't know of any open hardware, open software projects who try to do augmented reality right now. Of course, there's research. Research is more or less open most of the time, but research prototypes right now, while they are good at what they do, they are hardly like consumer products. So you can't really mass produce them and distribute them to the people. So yeah, you probably could do it, but I don't see currently I don't see any movement in this regard. Yeah. I shouldn't do that, but just for the previous question, I would suggest. Speak up, please. I shouldn't do that, but to answer briefly the previous question, WebAR and some effort from Mozilla are trying to bring at least the highest level of the stack in an open source fashion for augmented reality. Okay. Yeah, I was wondering for the example you showed on your research in your own lab about information retrieval. Those examples that are able to distinguish between one object and another, let's say an apple and a pear or a tomato works well, but I'm wondering how scalable it is because when you get to a morphicized example. Yeah, that's a good point because like what we were using was an actual food database for that. So it was very easy to classify like the vegetables because they are food. But if you have a database with everything in it, of course, it's a problem because like classification will get you some weird results. So maybe the vegetable would be recognized. I don't know as a lamp or something like that. Yeah, that's that's a problem. But this is only a problem like in this particular research case, it was more like a proof of concept to actually show. Okay, how can we use augmented reality in some more practical everyday scenarios. It wasn't really much focused on to find the perfect technical solution for providing like the classification of the images. This is also why we use an existing service for that and we didn't build our own. So we just use what was available. Okay, unless someone is beaming himself up from the Internet, then we had all the questions here answered. Thank you very, very much. Thank you. I hope to see you soon.
And his talk provides us an overview on AR in general and explains its possible uses for goods been a lot of talk about virtual reality like really influence our everyday lives that much. first one is that virtual reality applications the so-called mixed reality continuum, which will have really the power to revolutionize our for usage outside of the lab with a wonderful can we use it outdoors as well, which is very influence on our lives, we will still benefit So today's technology grows more complex and it's AR can really help with understanding technology content is projected onto. And there are even with it, which more or less could be quite the question of privacy and data security. There more sensitive data than current smartphones. So common for people to communicate with reach our destination? There has been research could have a benefit in our life, but that there world, which is totally fake. And the real world basically. This is also an important thing. The It's really the same question with the use cases.