Mycroft: A.I. in the desktop
Larrosa, Antonio
Control your desktop with your voice Mycroft is an open source virtual assistant similar to the ones provided on phones by some companies. It provides some interesting functionality, it's open source, it's easily expanded with new functionality and it's fun. During the last hackweek, I packaged it so it's easy to install on Tumbleweed and in this talk I'll try to show how it works, the changes I did to it in order to package it and maybe how to add a new skill.
 Yeah, now, right? Thanks. So, I was saying thanks for coming. We're going to start now and I'm going to talk about Minecraft, but before that, probably I should talk about who am I. I'm Antlar on IRC. I'm A. La Rosa on... Okay, just one second because Minecraft is running in the background and I will just make it not listen. Okay, I'm A. La Rosa on GroupWise. I've been a KD developer since the 97 and currently the president of KD Spain, which is a local organization related to KD. I've been an assistant user since 98 and a city user since 2004 for my previous company, but then I changed the company and I've been an assistant employee since 2013. If you are also an assistant employee, probably you don't know me because I'm working from home in Malaga in Spain. So, that's the reason I also don't know many of you because we mostly only meet during these conferences. And currently I started working at the OpenSUSE department, but I'm currently at the desktop team. In general, I'm a bit of an audiophile and music geek in general. My first application for KDE was MIDI player, for example, and I've been working with audio and I have some projects that I haven't talked about yet. So, probably if you are interested in music, then we can talk about that later. But now you're going to talk about Minecraft. Minecraft, there's a table here. How many of you know Minecraft? Know what it is? And how many of you have seen it working? Nobody. Perfect. Cool. At the end, I will give a demo. Okay. Let's hope it works. Well, Minecraft is an open source virtual assistant. Something similar to Siri or Google Assistant, things like that, but it's completely open source. It's mainly developed by Microsoft AI, Incorporated. It's a company. But this company is working on it and mainly they do one matching, something similar to Amazon Echo or Google Home, things like that. It's a small matching with eyes and a mouth. It's called Mark 1. This is a computer generated image, but this is a real picture of what the machine looks like. For those of you who are interested in what's inside, this is what's inside. Well, recently, this was created some years ago and recently they did a crowd funding on Kickstarter and developed this Mark 2, which is a bit more modern. The crowd funding already finished, but I think you can still buy this somewhere. Okay. So this company doesn't only develop the machines, these are small things, but they also develop the software for this. And this is called Microsoft Core. You can find it on GitHub. It uses a patch 2.0 license. It's written in Python. Originally, it was written in Python 2, of course. And recently they parted it to Python 3. I will talk more about that later. And, well, it has lots of Python dependencies. Okay. Really a lot. I think there were more like something like 50 something dependencies, direct dependencies. Okay. So you can imagine that this is a bit difficult to package. And the only small problem that I have with this, this Microsoft Core thing, is that you have to sign a CLA to submit code. And, well, it's something that you have to go through in order to submit source and collaborate with these people. But I think they are really free software. So, yeah, it's okay. Well, Microsoft Core is composed of four demons. Microsoft Message Bus, Microsoft Skills, Microsoft Audio, and Microsoft Speech Client. The first one is a demon that just do the messaging between the other demons. Just do the intercommunication between demons. Microsoft Skills is the demon that really does the work. Okay. And I will talk about Skills later. Microsoft Audio is a skill that converts speech to text so that it can listen to what I'm saying and translate that to text. Then it communicates through Message Bus to the Microsoft Skills demon. It does whatever has to be done. And then Microsoft Speech Client is the demon that converts text to speech so that it returns some output to the user. Okay. Well, apart from Microsoft Core, there's something called PlasmaMicroft, which is a Plasma widget developed by Aditya Mera. He's one of the KDE developers. He's a very nice guy. And if you have any problem with PlasmaMicroft or want to help him, because right now it's a one-man project, please talk to him because he's a really nice person. Okay. So let's talk about Skills. A skill is a component inside Microsoft that implements a skill. A skill is something that recognizes an intention from the user, optionally requests more information to the user if there's something missing that the skills need. It performs an action and then gives a reply to the user. Okay. All this is done by a skill. And I have an example here. But I think it's better to just go through the talk. And if at the end we have time, please remind me that we go and see the example of how it's done so that you can develop your own skills. And it's really, really easy and it's very, I mean, something that I recommend you to do. Yesterday, by the way, Takashi asked me how smart is Minecraft? Well, it's as smart as you develop your skills to be. Okay. Well, the skills work surprisingly well. I mean, it's too simple, I think, but it works really, really well and gives a lot of flexibility, as we will see now. Okay. But looking at the code, well, there are some things that could be improved, let's say. But you know what they say. Code is like messages, so it's better not to look how it's made. And so let's see how it works. And the sooner we get there, the better. So the first thing that I want to talk about is installation. How do you install? Because I guess by now you are asking yourself how to install Minecraft? Well, it can be installed in different ways. But the first one, the official one, is cloning the code from GitHub, writing some scripts as root, which copies the code to use Python, then it uses pip to install a lot of dependencies. And that's a mess on your system, basically. So the easy way is to use the packages for open source and the number width that I created. And I guess as soon as I have time, I will also try to test them on lib15. For lib42, it wasn't possible, because there were too many dependencies, but for lib15, I think it won't be a problem at all. Okay. Now I have to do a disclaimer that, of course, I'm not a micro developer, I only package it. So if anything works, it's because of the micro developers and if something breaks, it's probably because of my packaging. Okay. And why do I say this? Basically, because while packaging Microsoft, I had to do a lot of changes to the code. And for example, Microsoft by default uses a lot slash OPT to store credentials, to store catches for the user, mainly everything to store the skills, the code from the skills. And we cannot accept that inside open source, so I changed that to use the user home directory and submitted the code to the developers from Microsoft. Recently, they rejected my submission, because they accepted another submission that changes that to a slash bar, which is a bit better, but not really much better. But I have to adapt these changes. I forgot to say that all of the, well, most of the micro packaging, I did that on November last year, while on Hackweek. That, in fact, gives me some problems, because I had to sign the CLA in order to contribute code to the micro developers, but I was working on this during Hackweek, so it was a sucess development in some way, so I had to ask Ciaran, by the way, do you know if Ciaran is in the conference? I guess not. Okay. Well, it wasn't a problem at the end, but for a moment, I was a bit scared that I couldn't contribute it. Well, also, from the beginning, I used Python 3 for Minecraft, and at the beginning, in November last year, it wasn't supported, so I had to do many changes to support Python 3. I had to port some of the dependencies to Python 3. I had to do many changes on the skills, because the skills, I didn't say before, the skills, when you install a skill, the way to install a skill is to clone the skill repository from Jethub directly to your home directory. So patching that when the skill author is not very responsive, it's a bit difficult, but I found a way to patch things as soon as they were cloning. It's a kind of mess that works. Unfortunately, I think last week, they already ported the main development branch from Minecraft to Python 3, so I hope this gets better and better soon. This code, by the way, the code that I will show now, is based on the Minecraft code from a couple of weeks ago, so it's not using really, really the last version from Minecraft, but it has patches from upstream, and it works. Okay, another thing I changed was the way to run the four demos that I talked about before, because the official way to run them was to just run some scripts that are called startMicrofsh, and the user has to run that. So instead of that, I created some systemd user target and user services, so you can start them from systemd, and also change the way the login works, so that instead of using bar log, it uses journal CTL to look at the logs, which is something, by the way, that I recommend to do, if something fails, because it gives a lot of information. Okay, another thing I changed that is related to the first point is creating a default user configuration, the first time that the systemd services are run, because the way I changed the configuration, sorry, the way the directory is where Minecraft is installed, is to look first at the home of the configuration from the home directory, and if it cannot find the configuration there, then it looks at OPT as a fallback. So it's important to run this, and also it's important to download the full set of skills the first time that you run Minecraft, so that you have something that the Minecraft can do. Okay, so there's a warning, everything is in development, and demo time. Let's see. Okay, so this is the Minecraft Plasma widget, this is Plasma Minecraft, you can install it here, for example, on the bar, and then when you open it, this is something from before. Okay, when you open it, you can see a default information, the news, it recognizes our location using IP geolocation, so it takes the news in check. And from here, you can start and stop Minecraft. Okay. Right now, it's already started, and it's muted, but I will let him here now. And let's see. We have been trying to make this work in a nice way, but at the end, what I'm going to do is talk directly to the microphone in the laptop. Okay. So, hey, Minecraft, how do you do? I am doing well, thank you. Okay, so it recognizes what I'm saying and connects to the micro server, sometimes it recognizes too much. The hot word, this hey, micro thing, is recognized locally, but the rest of what you say is recorded and sent to the micro server, where it anonymizes it and then sends to Google, so by default, it's using Google to do the speak to test, which is the reason that it works so well. There's options to use other things, for example, using podcast things locally, and I tested it, and at least for me, it doesn't recognize mostly anything I say. There's an option also to use the deep speech thing from Mozilla that is currently being developed, but that's in development, and well, for now, it doesn't work very well, but I hope it gets better and better. Let's try something else. Hey, Minecraft, what's 2 plus 2? 4. I can do math. Hey, Minecraft, how high is the AFL tower? 1,063 feet city rang, first national rang, first world rang. Depending on the scale that gets the text, the request, then it works better or worse, and sometimes it gets the information from Wolfram Alpha as the calculation that we did before. Sometimes it uses Wikipedia as this thing. It can do many kinds of things, for example, let's see, I have here a list so that I don't forget anything. Hey, Minecraft, what's the capital of Spain? 2. Madrid is the capital of Spain and the largest municipality in both the community of Madrid and Spain as a whole. Hey, Minecraft, tell me a joke. Java, right once, run away. Well, let's see what else can we do. There are some skills that not only give an answer, but leave something running in the background, for example. Hey, Minecraft, play the news. Here is the latest news. Okay, this connects to our radio streaming service. Support for this in our podcast and the following message come from the UPX tour. Who knows that small business owners can't afford to take days off. Okay, it stops. There's something, for example, something interesting, like asking Minecraft to, for example, okay, let's do something nice. Hey, Minecraft, play Space Oddity by David Bowie from YouTube. Searching YouTube for Space Oddity by David Bowie. I don't know if it's playing. Somehow, right? Hey, Minecraft, stop. Okay. Hey, Minecraft, when was Alan Parsons born? For example. No, didn't get it. Friday, August 9th, 1968. Okay, that gets the information from Wikipedia also. Of course, I don't know who that is at all. You can also write your request here instead of speaking. If for some reason it doesn't understand something. Okay, so I'm going to show probably, you know, the interface from Plasma Minecraft, which is the nice thing here. This is a skill tab that currently doesn't work. That should show the skills that you have installed on the system. I mean, you have to think that this doesn't work. I think it's because of my packaging. It's not looking at the skills in the directory that they think they should be. But it's not necessarily really to make it work right now. Then you have a setting tab that you probably shouldn't touch. Because it allows the Plasma Minecraft widget to run and stop Minecraft in different... I'm not talking about him. It allows to stop and start Minecraft when you click on that button in different ways, depending on the installation way that you do it. But if you are using the Minecraft packages, then that's the only thing that you have to do there. And then this is interesting, is the skills tab which shows all the skills that are available and allows you to install them. As I said before, it's not currently the recognition of what the skills are installed and what the skills are not installed. It's not working. So some of these are already installed. And they're not recognized yet. But you can see here how it works. For example, let me search for something that I know that is not installed, the Hue skill that controls Philips Hue lights. Okay. You can click on Readme. Of course, it's a heavy GitHub user. So it goes directly to the GitHub page for the skill. And there you can see the Readme, which usually for, I think, all of the skills says how to install them, which basically is clicking on Install on that widget before. Some of them require a configuration. In general, it's not necessary to configure anything. And then it gives you the sentences that these skills understand. Okay. In this case, turn on my lights, set lights scene or something. In general, when you install a skill, as I said, my graph clones the repository from the skill into a skills directory. Okay. Oh, no. You can see that. That's better, right? Okay. So those are the skills that I have here. And as I said, the skills are cloned there. And then usually it uses PIP to install the dependencies of that skill. Since I remove all PIP usage from my graph, currently, if the skills you're installing is using some strange dependency that you don't have in your system, it just won't work. It doesn't do anything. Apart from logging in your logs that these skills need these requirements, these dependencies, so please be sure they install on your system. So that's what I said, that you should run Journal CTL and have a look at the user unit, my graph skills. Okay. I'm going to do that. Where you can see a lot of table information. As I said, this is mainly in development. If we go to the beginning, we can find some interesting things here. I think it's mostly time to end the talk, but I have to show something else yet. Hey, Minecraft. Remind me in four minutes to end the talk. You have a new reminder on three minutes and 59 seconds from now. To be sure that I give a couple of minutes at least for questions. And remove this. When the skills registers to the Microsoft skill demo, the skill can register a new vocabulary so that Microsoft recognizes more things. In this case, for example, we can see, let's do a better example. Where do I have this? I'm going to find it right now. I will show you just the example. Hey, Minecraft. Open Dolphin. So Dolphin is something that is not really, I mean, it doesn't exist in the skill, but the skill gets the list of applications that you have installed on your system and registers them as vocabulary. Then I can say, hey, Minecraft. Close Dolphin. I don't know what that means. Hey, Minecraft. Close Dolphin. Close. This is one of those things, by the way, that I said before that it's better not to look how it works. Because currently, the skills, when you ask it to close something, it just do an OS system, kill, all whatever. So, yeah. I hope it was a decision to do it quick so that it can improve later. I hope it's soon. What else do I want to show? I don't know if you know some nice thing about KDE that is the activities. You can have activities for different usage of your computer. So you can say, hey, Minecraft. Show activities. It shows the activities from the desktop. I don't have anything that probably, if I say, news and articles right now, it will just recognize it wrong and doesn't recognize that I'm referring to that. But in general, you can switch to another activity and control your computer with this. Something nice that I didn't show. Let's see. Let me run Kate. Hey, Minecraft. Type this is a test. Type in your request. You can use Minecraft to dictate things. And of course, if you speak English better than me, it will recognize you better than me. Let's see if I have something else that I want to show. This is nice. Hey, Minecraft, what's the weather like? Right now, it's scattered clouds and 21 degrees for a high of 24 and a low of 16. And if you look at, there are no windows. Cool. So you cannot check if it's right or not. Hey, Minecraft, will it rain today? Light rain is expected on Monday. So we can infer that. No. Hey, Minecraft, increase volume. There is a reminder to end the talk. Tell me to end reminder to stop the current reminder. Hey, Minecraft. Hey, Minecraft, stop the reminder. The reminder has been shut off. Okay. And what I was saying before. Hey, Minecraft, increase volume. And it was already at the max. But you see how it works. I don't know. Do you want to ask questions? Do you prefer to... I show you how our skill is developed quite quickly. Yeah. Wait, wait. Maybe I can get you here. This. How this works. Here. No. Okay. I have one question. Currently, there is only in English. Is it possible to locate two different languages? Yes, it's possible. And currently, English is the best supported language, of course. But if you go to... Did I close the console window? Yes. If you go to... Let me change this first. To Minecraft skills, for example... I don't know. Let me see the Wikipedia skill. Okay. I will take the opportunity to show a bit how the skills work. There is a set of directories here inside each of the skills. They define the vocabulary that the skill understands. Okay. So if you go inside this vocabulary, then there is a different directory for each of the languages. And you can go to English and you can see here that this skill understands when you mention Wikipedia, when you say, tell me about... So you can say, hey, Minecraft, tell me about London. Let me look up London. Okay. It says London is the capital and most popular citizen of the world. It says that for Wikipedia because the skills recognizer tells me about... The southeast of the island of Great Britain, London has been a major settlement for two years. And then you can also define other languages there. Right now, there is some basic support for German, for Italian, for French, I think. I think there was some Swedish support also. You have to think that the speech-to-text thing is done by Google. So it basically works in the language. And the text-to-speech is currently done with an application that the developers did, that is called Mimic. But you can configure it to also use Google to do that. So it should work. Okay. And the only thing that you have to do to support other languages is to implement all the vocabulary for all the skills. Not only for the recognition of the words, but also for the dialogue, for what Microsoft responds to the user, the Microsoft replies to the user. So in this case, for example, no entry phone, it has different sentences. That means there was no entry phone on Wikipedia about that subject. So that doesn't always always the same thing. But it just gets them randomly. And then there's also a regular expression that some skills have. That's a regular expression so that you can say different things and get which part of that sentence is the article that it should be looked for. Okay. And is there another question? Did I answer your question? Yeah, thank you. And I have also one question. When I asked to delete my disk, what did happen? That depends on the skills of the developer that developed the skill. Yeah. Of course, if you have a skill that recognizes delete my hard disk and you installed it, then yeah, it's partially your fault somehow. But yeah, in that case, you have to be careful with what you install, of course. Yeah. Which is something that probably should be thought about because right now the Microsoft developers mainly care about the machines that they have so that they don't care about, for example, installing credentials for the user on the home directory. And right now, as I said, they were shared on the shared directory. And if you have two users on the systems, well, they just accept changes and accept patches for related to the desktop. But most of the skills related to the desktop are done by Aditya Mera that I mentioned before. Yeah. Yeah. Installing skills is something, yeah. You have to be careful. And no questions? Any other questions? Yeah. Okay. First, thanks for all the work you're doing here because I know from my own experience how hard it can be to package such complex stuff. And even better, that you're on stream things because that makes it also easier. Now I'm also looking forward to integrate this or also other tools which are now emerging like the Susie project from Fossasia and other desktops and other distros. So thanks a lot. Thank you. In fact, I should say that I did the packaging, but Thomas Vatal and Thomas Bertol, I don't know if he's here, by the way, another developer, other developer languages Python maintainers also helped me a lot because they were, well, they had to review a lot of packages and they made me fix many things in those packages. So they also did a lot of work. Yeah. And before I finish, where's this? Let me say thanks. And I have a blog which currently has one entry that is related to Minecraft. And it explains how to install this on Open Susie at Amberweed. And I did the blog post two months ago so it doesn't work anymore because, finally, most of the dependencies were already in factory which they weren't on when I wrote that, but I plan to update it soon so you can see my second post there. Yeah. There's no other question. If there are more questions, you can ask me outside, probably. And thank you for coming.
 Yeah, now, right? Thanks. So, I was saying thanks for coming. We're going to start now  and I'm going to talk about Minecraft, but before that, probably I should talk about  who am I. I'm Antlar on IRC. I'm A. La Rosa on... Okay, just one second because  Minecraft is running in the background and I will just make it not listen. Okay, I'm  A. La Rosa on GroupWise. I've been a KD developer since the 97 and currently  the president of KD Spain, which is a local organization related to KD. I've been  an assistant user since 98 and a city user since 2004 for my previous company, but  then I changed the company and I've been an assistant employee since 2013. If you  are also an assistant employee, probably you don't know me because I'm working from  home in Malaga in Spain. So, that's the reason I also don't know many of you  because we mostly only meet during these conferences. And currently I started  working at the OpenSUSE department, but I'm currently at the desktop team.  In general, I'm a bit of an audiophile and music geek in general. My first application  for KDE was MIDI player, for example, and I've been working with audio and I have  some projects that I haven't talked about yet. So, probably if you are interested  in music, then we can talk about that later. But now you're going to talk about  Minecraft. Minecraft, there's a table here. How many of you know Minecraft? Know what  it is? And how many of you have seen it working? Nobody. Perfect. Cool. At the end,  I will give a demo. Okay. Let's hope it works. Well, Minecraft is an open source  virtual assistant. Something similar to Siri or Google Assistant, things like that,  but it's completely open source. It's mainly developed by Microsoft AI,  Incorporated. It's a company. But this company is working on it and mainly they  do one matching, something similar to Amazon Echo or Google Home, things like  that. It's a small matching with eyes and a mouth. It's called Mark 1. This is a  computer generated image, but this is a real picture of what the machine looks  like. For those of you who are interested in what's inside, this is what's inside.  Well, recently, this was created some years ago and recently they did a crowd  funding on Kickstarter and developed this Mark 2, which is a bit more modern.  The crowd funding already finished, but I think you can still buy this somewhere.  Okay. So this company doesn't only develop the machines, these are small things,  but they also develop the software for this. And this is called Microsoft Core.  You can find it on GitHub. It uses a patch 2.0 license. It's written in Python.  Originally, it was written in Python 2, of course. And recently they parted it to  Python 3. I will talk more about that later. And, well, it has lots of Python  dependencies. Okay. Really a lot. I think there were more like something like 50  something dependencies, direct dependencies. Okay. So you can imagine that this is a  bit difficult to package. And the only small problem that I have with this,  this Microsoft Core thing, is that you have to sign a CLA to submit code. And, well,  it's something that you have to go through in order to submit source and  collaborate with these people. But I think they are really free software.  So, yeah, it's okay. Well, Microsoft Core is composed of four demons.  Microsoft Message Bus, Microsoft Skills, Microsoft Audio, and Microsoft Speech Client.  The first one is a demon that just do the messaging between the other demons.  Just do the intercommunication between demons. Microsoft Skills is the demon that  really does the work. Okay. And I will talk about Skills later.  Microsoft Audio is a skill that converts speech to text so that it can listen to  what I'm saying and translate that to text. Then it communicates through  Message Bus to the Microsoft Skills demon. It does whatever has to be done.  And then Microsoft Speech Client is the demon that converts text to speech so  that it returns some output to the user. Okay. Well, apart from Microsoft Core,  there's something called PlasmaMicroft, which is a Plasma widget developed by  Aditya Mera. He's one of the KDE developers. He's a very nice guy. And if you have any  problem with PlasmaMicroft or want to help him, because right now it's a one-man  project, please talk to him because he's a really nice person.  Okay. So let's talk about Skills. A skill is a component inside Microsoft that  implements a skill. A skill is something that recognizes an intention from the  user, optionally requests more information to the user if there's  something missing that the skills need. It performs an action and then gives a  reply to the user. Okay. All this is done by a skill. And I have an example here.  But I think it's better to just go through the talk. And if at the end we have  time, please remind me that we go and see the example of how it's done so that  you can develop your own skills. And it's really, really easy and it's very,  I mean, something that I recommend you to do. Yesterday, by the way, Takashi  asked me how smart is Minecraft? Well, it's as smart as you develop your skills  to be. Okay. Well, the skills work surprisingly well. I mean, it's too  simple, I think, but it works really, really well and gives a lot of flexibility,  as we will see now. Okay. But looking at the code, well, there are some things  that could be improved, let's say. But you know what they say. Code is like  messages, so it's better not to look how it's made. And so let's see how it works.  And the sooner we get there, the better. So the first thing that I want to talk  about is installation. How do you install? Because I guess by now you are asking  yourself how to install Minecraft? Well, it can be installed in different ways.  But the first one, the official one, is cloning the code from GitHub,  writing some scripts as root, which copies the code to use Python,  then it uses pip to install a lot of dependencies. And that's a mess on your  system, basically. So the easy way is to use the packages for open source  and the number width that I created. And I guess as soon as I have time, I will also  try to test them on lib15. For lib42, it wasn't possible, because there were too  many dependencies, but for lib15, I think it won't be a problem at all.  Okay. Now I have to do a disclaimer that, of course, I'm not a  micro developer, I only package it. So if anything works, it's because of the  micro developers and if something breaks, it's probably because of my packaging.  Okay. And why do I say this? Basically, because while packaging  Microsoft, I had to do a lot of changes to the code. And for example,  Microsoft by default uses a lot slash OPT to store credentials, to store  catches for the user, mainly everything to store the skills, the code from the  skills. And we cannot accept that inside open source, so I changed that to use the  user home directory and submitted the code to the developers from Microsoft.  Recently, they rejected my submission, because they accepted another submission  that changes that to a slash bar, which is a bit better, but not really much  better. But I have to adapt these changes. I forgot to say that all of the,  well, most of the micro packaging, I did that on November last year, while on  Hackweek. That, in fact, gives me some problems, because I had to sign the  CLA in order to contribute code to the micro developers, but I was working on  this during Hackweek, so it was a sucess development in some way, so I had to ask  Ciaran, by the way, do you know if Ciaran is in the conference? I guess not.  Okay. Well, it wasn't a problem at the end, but for a moment, I was a bit scared  that I couldn't contribute it. Well, also, from the beginning, I used Python 3  for Minecraft, and at the beginning, in November last year, it wasn't  supported, so I had to do many changes to support Python 3. I had to port some of  the dependencies to Python 3. I had to do many changes on the skills, because  the skills, I didn't say before, the skills, when you install a skill, the way to  install a skill is to clone the skill repository from Jethub directly to your  home directory. So patching that when the skill author is not very responsive, it's  a bit difficult, but I found a way to patch things as soon as they were  cloning. It's a kind of mess that works. Unfortunately, I think last week, they  already ported the main development branch from Minecraft to Python 3, so I  hope this gets better and better soon. This code, by the way, the code that I will  show now, is based on the Minecraft code from a couple of weeks ago, so it's not  using really, really the last version from Minecraft, but it has patches from  upstream, and it works. Okay, another thing I changed was the way to run the four  demos that I talked about before, because the official way to run them was to just  run some scripts that are called startMicrofsh, and the user has to run  that. So instead of that, I created some systemd user target and user services, so  you can start them from systemd, and also change the way the login works, so that  instead of using bar log, it uses journal CTL to look at the logs, which is  something, by the way, that I recommend to do, if something fails, because it gives  a lot of information. Okay, another thing I changed that is related to the first  point is creating a default user configuration, the first time that the  systemd services are run, because the way I changed the configuration, sorry, the  way the directory is where Minecraft is installed, is to look first at the home  of the configuration from the home directory, and if it cannot find the  configuration there, then it looks at OPT as a fallback. So it's important to run  this, and also it's important to download the full set of skills the first time  that you run Minecraft, so that you have something that the Minecraft can do. Okay,  so there's a warning, everything is in development, and demo time.  Let's see. Okay, so this is the Minecraft Plasma widget, this is Plasma  Minecraft, you can install it here, for example, on the bar, and then when you open  it, this is something from before. Okay, when you open it, you can see a default  information, the news, it recognizes our location using IP geolocation, so it takes  the news in check. And from here, you can start and stop Minecraft. Okay. Right now,  it's already started, and it's muted, but I will let him here now. And let's see.  We have been trying to make this work in a nice way, but at the end, what I'm going  to do is talk directly to the microphone in the laptop. Okay. So, hey, Minecraft, how  do you do? I am doing well, thank you. Okay, so it recognizes what I'm saying and connects  to the micro server, sometimes it recognizes too much. The hot word, this hey, micro thing,  is recognized locally, but the rest of what you say is recorded and sent to the micro  server, where it anonymizes it and then sends to Google, so by default, it's using Google  to do the speak to test, which is the reason that it works so well. There's options to  use other things, for example, using podcast things locally, and I tested it, and at least  for me, it doesn't recognize mostly anything I say. There's an option also to use the deep  speech thing from Mozilla that is currently being developed, but that's in development,  and well, for now, it doesn't work very well, but I hope it gets better and better. Let's  try something else. Hey, Minecraft, what's 2 plus 2?  4. I can do math. Hey, Minecraft, how high is the AFL tower?  1,063 feet city rang, first national rang, first world rang.  Depending on the scale that gets the text, the request, then it works better or worse,  and sometimes it gets the information from Wolfram Alpha as the calculation that we did  before. Sometimes it uses Wikipedia as this thing. It can do many kinds of things, for  example, let's see, I have here a list so that I don't forget anything. Hey, Minecraft, what's  the capital of Spain?  2. Madrid is the capital of Spain and the largest municipality in both the community  of Madrid and Spain as a whole. Hey, Minecraft, tell me a joke.  Java, right once, run away. Well, let's see what else can we do. There are some skills  that not only give an answer, but leave something running in the background, for example. Hey,  Minecraft, play the news.  Here is the latest news.  Okay, this connects to our radio streaming service.  Support for this in our podcast and the following message come from the UPX tour. Who knows  that small business owners can't afford to take days off.  Okay, it stops. There's something, for example, something interesting, like asking Minecraft  to, for example, okay, let's do something nice.  Hey, Minecraft, play Space Oddity by David Bowie from YouTube.  Searching YouTube for Space Oddity by David Bowie.  I don't know if it's playing.  Somehow, right? Hey, Minecraft, stop.  Okay. Hey, Minecraft, when was Alan Parsons born?  For example. No, didn't get it.  Friday, August 9th, 1968.  Okay, that gets the information from Wikipedia also. Of course, I don't know who that is  at all. You can also write your request here instead of speaking. If for some reason it  doesn't understand something. Okay, so I'm going to show probably, you know, the interface  from Plasma Minecraft, which is the nice thing here.  This is a skill tab that currently doesn't work. That should show the skills that you  have installed on the system. I mean, you have to think that this doesn't work. I think  it's because of my packaging. It's not looking at the skills in the directory that they think  they should be. But it's not necessarily really to make it work right now. Then you have a  setting tab that you probably shouldn't touch. Because it allows the Plasma Minecraft widget  to run and stop Minecraft in different... I'm not talking about him. It allows to stop  and start Minecraft when you click on that button in different ways, depending on the  installation way that you do it. But if you are using the Minecraft packages, then that's  the only thing that you have to do there. And then this is interesting, is the skills  tab which shows all the skills that are available and allows you to install them. As I said before,  it's not currently the recognition of what the skills are installed and what the skills  are not installed. It's not working. So some of these are already installed. And they're  not recognized yet. But you can see here how it works. For example, let me search for something  that I know that is not installed, the Hue skill that controls Philips Hue lights. Okay.  You can click on Readme. Of course, it's a heavy GitHub user. So it goes directly to  the GitHub page for the skill. And there you can see the Readme, which usually for, I think,  all of the skills says how to install them, which basically is clicking on Install on that  widget before. Some of them require a configuration. In general, it's not necessary to configure  anything. And then it gives you the sentences that these skills understand. Okay. In this  case, turn on my lights, set lights scene or something. In general, when you install  a skill, as I said, my graph clones the repository from the skill into a skills directory. Okay.  Oh, no. You can see that. That's better, right? Okay. So those are the skills that I have  here. And as I said, the skills are cloned there. And then usually it uses PIP to install  the dependencies of that skill. Since I remove all PIP usage from my graph, currently, if  the skills you're installing is using some strange dependency that you don't have in  your system, it just won't work. It doesn't do anything. Apart from logging in your logs  that these skills need these requirements, these dependencies, so please be sure they  install on your system. So that's what I said, that you should run Journal CTL and have a  look at the user unit, my graph skills. Okay. I'm going to do that. Where you can see a  lot of table information. As I said, this is mainly in development. If we go to the  beginning, we can find some interesting things here. I think it's mostly time to end the  talk, but I have to show something else yet. Hey, Minecraft. Remind me in four minutes  to end the talk. You have a new reminder on three minutes and 59 seconds from now.  To be sure that I give a couple of minutes at least for questions. And remove this. When  the skills registers to the Microsoft skill demo, the skill can register a new vocabulary  so that Microsoft recognizes more things. In this case, for example, we can see, let's  do a better example. Where do I have this? I'm going to find it right now. I will show  you just the example. Hey, Minecraft. Open Dolphin. So Dolphin is something that is not  really, I mean, it doesn't exist in the skill, but the skill gets the list of applications  that you have installed on your system and registers them as vocabulary. Then I can say,  hey, Minecraft. Close Dolphin. I don't know what that means. Hey, Minecraft. Close Dolphin.  Close. This is one of those things, by the way, that I said before that it's better not  to look how it works. Because currently, the skills, when you ask it to close something,  it just do an OS system, kill, all whatever. So, yeah. I hope it was a decision to do it  quick so that it can improve later. I hope it's soon. What else do I want to show? I  don't know if you know some nice thing about KDE that is the activities. You can have activities  for different usage of your computer. So you can say, hey, Minecraft. Show activities.  It shows the activities from the desktop. I don't have anything that probably, if I say,  news and articles right now, it will just recognize it wrong and doesn't recognize that  I'm referring to that. But in general, you can switch to another activity and control  your computer with this. Something nice that I didn't show. Let's see. Let me run Kate.  Hey, Minecraft. Type this is a test. Type in your request. You can use Minecraft to dictate  things. And of course, if you speak English better than me, it will recognize you better  than me. Let's see if I have something else that I want to show. This is nice. Hey, Minecraft,  what's the weather like? Right now, it's scattered clouds and 21 degrees for a high of 24 and  a low of 16. And if you look at, there are no windows. Cool. So you cannot check if it's  right or not. Hey, Minecraft, will it rain today? Light rain is expected on Monday. So  we can infer that. No. Hey, Minecraft, increase volume. There is a reminder to end the talk.  Tell me to end reminder to stop the current reminder. Hey, Minecraft. Hey, Minecraft, stop  the reminder. The reminder has been shut off. Okay. And what I was saying before. Hey, Minecraft,  increase volume. And it was already at the max. But you see how it works. I don't know.  Do you want to ask questions? Do you prefer to... I show you how our skill is developed  quite quickly. Yeah. Wait, wait. Maybe I can get you here. This. How this works. Here.  No. Okay. I have one question. Currently, there is only in English. Is it possible to  locate two different languages? Yes, it's possible. And currently, English is the best  supported language, of course. But if you go to... Did I close the console window? Yes.  If you go to... Let me change this first. To Minecraft skills, for example... I don't  know. Let me see the Wikipedia skill. Okay. I will take the opportunity to show a bit  how the skills work. There is a set of directories here inside each of the skills. They define  the vocabulary that the skill understands. Okay. So if you go inside this vocabulary,  then there is a different directory for each of the languages. And you can go to English  and you can see here that this skill understands when you mention Wikipedia, when you say,  tell me about... So you can say, hey, Minecraft, tell me about London. Let me look up London.  Okay. It says London is the capital and most popular citizen of the world. It says that  for Wikipedia because the skills recognizer tells me about...  The southeast of the island of Great Britain, London has been a major settlement for two  years. And then you can also define other languages there. Right now, there is some basic support  for German, for Italian, for French, I think. I think there was some Swedish support also.  You have to think that the speech-to-text thing is done by Google. So it basically works  in the language. And the text-to-speech is currently done with an application that the  developers did, that is called Mimic. But you can configure it to also use Google to  do that. So it should work. Okay. And the only thing that you have to do to support other  languages is to implement all the vocabulary for all the skills. Not only for the recognition  of the words, but also for the dialogue, for what Microsoft responds to the user, the  Microsoft replies to the user. So in this case, for example, no entry phone, it has different  sentences. That means there was no entry phone on Wikipedia about that subject. So that doesn't  always always the same thing. But it just gets them randomly. And then there's also a regular  expression that some skills have. That's a regular expression so that you can say different  things and get which part of that sentence is the article that it should be looked for.  Okay. And is there another question? Did I answer your question?  Yeah, thank you. And I have also one question. When I asked to delete my disk, what did  happen?  That depends on the skills of the developer that developed the skill. Yeah. Of course,  if you have a skill that recognizes delete my hard disk and you installed it, then yeah,  it's partially your fault somehow. But yeah, in that case, you have to be careful with  what you install, of course. Yeah. Which is something that probably should be thought  about because right now the Microsoft developers mainly care about the machines that they have  so that they don't care about, for example, installing credentials for the user on the  home directory. And right now, as I said, they were shared on the shared directory. And  if you have two users on the systems, well, they just accept changes and accept patches  for related to the desktop. But most of the skills related to the desktop are done by  Aditya Mera that I mentioned before. Yeah. Yeah. Installing skills is something, yeah.  You have to be careful. And no questions? Any other questions? Yeah.  Okay. First, thanks for all the work you're doing here because I know from my own experience  how hard it can be to package such complex stuff. And even better, that you're on stream  things because that makes it also easier. Now I'm also looking forward to integrate  this or also other tools which are now emerging like the Susie project from Fossasia and other  desktops and other distros. So thanks a lot. Thank you. In fact, I should say that I did  the packaging, but Thomas Vatal and Thomas Bertol, I don't know if he's here, by the  way, another developer, other developer languages Python maintainers also helped me a lot because  they were, well, they had to review a lot of packages and they made me fix many things  in those packages. So they also did a lot of work. Yeah.  And before I finish, where's this? Let me say thanks. And I have a blog which currently  has one entry that is related to Minecraft. And it explains how to install this on Open  Susie at Amberweed. And I did the blog post two months ago so it doesn't work anymore  because, finally, most of the dependencies were already in factory which they weren't  on when I wrote that, but I plan to update it soon so you can see my second post there.  Yeah. There's no other question. If there are more questions, you can ask me outside,  probably. And thank you for coming.
openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_0_155.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_156_467.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_468_659.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_660_815.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_816_875.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_876_923.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_924_1043.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_1044_4763.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_4764_4847.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_4848_5183.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_5184_9719.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_9720_20999.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_21000_21107.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_21108_21287.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_21288_22379.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_22380_32471.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_32472_32579.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_32580_32651.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_32652_33071.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_33072_33647.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_33648_34139.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_34140_35567.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_35568_35699.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_35700_36563.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_36564_37007.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_37008_37103.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_37104_37199.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_37200_37283.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_37284_37355.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_37356_37631.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_37632_38351.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_38352_39635.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_39636_40127.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_40128_40427.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_40428_40559.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_40560_43919.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_43920_44123.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_44124_44243.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_44244_44387.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_44388_46163.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_46164_47219.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_47220_53015.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_53016_53303.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_53304_53975.png openSUSE_Conference_2018_-_Mycroft__A_I__in_the_de_Q5YqWLJmg9A_53976_54096.png
Yeah, now, right? Thanks. So, I was saying thanks for coming. We're going to start now and I'm going to talk about Minecraft, but before that, probably I should talk about who am I. I'm Antlar on IRC. I'm A. La Rosa on... Okay, just one second because Minecraft is running in the background and I will just make it not listen. Okay, I'm A. La Rosa on GroupWise. I've been a KD developer since the 97 and currently the president of KD Spain, which is a local organization related to KD. I've been an assistant user since 98 and a city user since 2004 for my previous company, but then I changed the company and I've been an assistant employee since 2013. If you are also an assistant employee, probably you don't know me because I'm working from home in Malaga in Spain. So, that's the reason I also don't know many of you because we mostly only meet during these conferences. And currently I started working at the OpenSUSE department, but I'm currently at the desktop team. In general, I'm a bit of an audiophile and music geek in general. My first application for KDE was MIDI player, for example, and I've been working with audio and I have some projects that I haven't talked about yet. So, probably if you are interested in music, then we can talk about that later. But now you're going to talk about Minecraft. Minecraft, there's a table here. How many of you know Minecraft? Know what it is? And how many of you have seen it working? Nobody. Perfect. Cool. At the end, I will give a demo. Okay. Let's hope it works. Well, Minecraft is an open source virtual assistant. Something similar to Siri or Google Assistant, things like that, but it's completely open source. It's mainly developed by Microsoft AI, Incorporated. It's a company. But this company is working on it and mainly they do one matching, something similar to Amazon Echo or Google Home, things like that. It's a small matching with eyes and a mouth. It's called Mark 1. This is a computer generated image, but this is a real picture of what the machine looks like. For those of you who are interested in what's inside, this is what's inside. Well, recently, this was created some years ago and recently they did a crowd funding on Kickstarter and developed this Mark 2, which is a bit more modern. The crowd funding already finished, but I think you can still buy this somewhere. Okay. So this company doesn't only develop the machines, these are small things, but they also develop the software for this. And this is called Microsoft Core. You can find it on GitHub. It uses a patch 2.0 license. It's written in Python. Originally, it was written in Python 2, of course. And recently they parted it to Python 3. I will talk more about that later. And, well, it has lots of Python dependencies. Okay. Really a lot. I think there were more like something like 50 something dependencies, direct dependencies. Okay. So you can imagine that this is a bit difficult to package. And the only small problem that I have with this, this Microsoft Core thing, is that you have to sign a CLA to submit code. And, well, it's something that you have to go through in order to submit source and collaborate with these people. But I think they are really free software. So, yeah, it's okay. Well, Microsoft Core is composed of four demons. Microsoft Message Bus, Microsoft Skills, Microsoft Audio, and Microsoft Speech Client. The first one is a demon that just do the messaging between the other demons. Just do the intercommunication between demons. Microsoft Skills is the demon that really does the work. Okay. And I will talk about Skills later. Microsoft Audio is a skill that converts speech to text so that it can listen to what I'm saying and translate that to text. Then it communicates through Message Bus to the Microsoft Skills demon. It does whatever has to be done. And then Microsoft Speech Client is the demon that converts text to speech so that it returns some output to the user. Okay. Well, apart from Microsoft Core, there's something called PlasmaMicroft, which is a Plasma widget developed by Aditya Mera. He's one of the KDE developers. He's a very nice guy. And if you have any problem with PlasmaMicroft or want to help him, because right now it's a one-man project, please talk to him because he's a really nice person. Okay. So let's talk about Skills. A skill is a component inside Microsoft that implements a skill. A skill is something that recognizes an intention from the user, optionally requests more information to the user if there's something missing that the skills need. It performs an action and then gives a reply to the user. Okay. All this is done by a skill. And I have an example here. But I think it's better to just go through the talk. And if at the end we have time, please remind me that we go and see the example of how it's done so that you can develop your own skills. And it's really, really easy and it's very, I mean, something that I recommend you to do. Yesterday, by the way, Takashi asked me how smart is Minecraft? Well, it's as smart as you develop your skills to be. Okay. Well, the skills work surprisingly well. I mean, it's too simple, I think, but it works really, really well and gives a lot of flexibility, as we will see now. Okay. But looking at the code, well, there are some things that could be improved, let's say. But you know what they say. Code is like messages, so it's better not to look how it's made. And so let's see how it works. And the sooner we get there, the better. So the first thing that I want to talk about is installation. How do you install? Because I guess by now you are asking yourself how to install Minecraft? Well, it can be installed in different ways. But the first one, the official one, is cloning the code from GitHub, writing some scripts as root, which copies the code to use Python, then it uses pip to install a lot of dependencies. And that's a mess on your system, basically. So the easy way is to use the packages for open source and the number width that I created. And I guess as soon as I have time, I will also try to test them on lib15. For lib42, it wasn't possible, because there were too many dependencies, but for lib15, I think it won't be a problem at all. Okay. Now I have to do a disclaimer that, of course, I'm not a micro developer, I only package it. So if anything works, it's because of the micro developers and if something breaks, it's probably because of my packaging. Okay. And why do I say this? Basically, because while packaging Microsoft, I had to do a lot of changes to the code. And for example, Microsoft by default uses a lot slash OPT to store credentials, to store catches for the user, mainly everything to store the skills, the code from the skills. And we cannot accept that inside open source, so I changed that to use the user home directory and submitted the code to the developers from Microsoft. Recently, they rejected my submission, because they accepted another submission that changes that to a slash bar, which is a bit better, but not really much better. But I have to adapt these changes. I forgot to say that all of the, well, most of the micro packaging, I did that on November last year, while on Hackweek. That, in fact, gives me some problems, because I had to sign the CLA in order to contribute code to the micro developers, but I was working on this during Hackweek, so it was a sucess development in some way, so I had to ask Ciaran, by the way, do you know if Ciaran is in the conference? I guess not. Okay. Well, it wasn't a problem at the end, but for a moment, I was a bit scared that I couldn't contribute it. Well, also, from the beginning, I used Python 3 for Minecraft, and at the beginning, in November last year, it wasn't supported, so I had to do many changes to support Python 3. I had to port some of the dependencies to Python 3. I had to do many changes on the skills, because the skills, I didn't say before, the skills, when you install a skill, the way to install a skill is to clone the skill repository from Jethub directly to your home directory. So patching that when the skill author is not very responsive, it's a bit difficult, but I found a way to patch things as soon as they were cloning. It's a kind of mess that works. Unfortunately, I think last week, they already ported the main development branch from Minecraft to Python 3, so I hope this gets better and better soon. This code, by the way, the code that I will show now, is based on the Minecraft code from a couple of weeks ago, so it's not using really, really the last version from Minecraft, but it has patches from upstream, and it works. Okay, another thing I changed was the way to run the four demos that I talked about before, because the official way to run them was to just run some scripts that are called startMicrofsh, and the user has to run that. So instead of that, I created some systemd user target and user services, so you can start them from systemd, and also change the way the login works, so that instead of using bar log, it uses journal CTL to look at the logs, which is something, by the way, that I recommend to do, if something fails, because it gives a lot of information. Okay, another thing I changed that is related to the first point is creating a default user configuration, the first time that the systemd services are run, because the way I changed the configuration, sorry, the way the directory is where Minecraft is installed, is to look first at the home of the configuration from the home directory, and if it cannot find the configuration there, then it looks at OPT as a fallback. So it's important to run this, and also it's important to download the full set of skills the first time that you run Minecraft, so that you have something that the Minecraft can do. Okay, so there's a warning, everything is in development, and demo time. Let's see. Okay, so this is the Minecraft Plasma widget, this is Plasma Minecraft, you can install it here, for example, on the bar, and then when you open it, this is something from before. Okay, when you open it, you can see a default information, the news, it recognizes our location using IP geolocation, so it takes the news in check. And from here, you can start and stop Minecraft. Okay. Right now, it's already started, and it's muted, but I will let him here now. And let's see. We have been trying to make this work in a nice way, but at the end, what I'm going to do is talk directly to the microphone in the laptop. Okay. So, hey, Minecraft, how do you do? I am doing well, thank you. Okay, so it recognizes what I'm saying and connects to the micro server, sometimes it recognizes too much. The hot word, this hey, micro thing, is recognized locally, but the rest of what you say is recorded and sent to the micro server, where it anonymizes it and then sends to Google, so by default, it's using Google to do the speak to test, which is the reason that it works so well. There's options to use other things, for example, using podcast things locally, and I tested it, and at least for me, it doesn't recognize mostly anything I say. There's an option also to use the deep speech thing from Mozilla that is currently being developed, but that's in development, and well, for now, it doesn't work very well, but I hope it gets better and better. Let's try something else. Hey, Minecraft, what's 2 plus 2? 4. I can do math. Hey, Minecraft, how high is the AFL tower? 1,063 feet city rang, first national rang, first world rang. Depending on the scale that gets the text, the request, then it works better or worse, and sometimes it gets the information from Wolfram Alpha as the calculation that we did before. Sometimes it uses Wikipedia as this thing. It can do many kinds of things, for example, let's see, I have here a list so that I don't forget anything. Hey, Minecraft, what's the capital of Spain? 2. Madrid is the capital of Spain and the largest municipality in both the community of Madrid and Spain as a whole. Hey, Minecraft, tell me a joke. Java, right once, run away. Well, let's see what else can we do. There are some skills that not only give an answer, but leave something running in the background, for example. Hey, Minecraft, play the news. Here is the latest news. Okay, this connects to our radio streaming service. Support for this in our podcast and the following message come from the UPX tour. Who knows that small business owners can't afford to take days off. Okay, it stops. There's something, for example, something interesting, like asking Minecraft to, for example, okay, let's do something nice. Hey, Minecraft, play Space Oddity by David Bowie from YouTube. Searching YouTube for Space Oddity by David Bowie. I don't know if it's playing. Somehow, right? Hey, Minecraft, stop. Okay. Hey, Minecraft, when was Alan Parsons born? For example. No, didn't get it. Friday, August 9th, 1968. Okay, that gets the information from Wikipedia also. Of course, I don't know who that is at all. You can also write your request here instead of speaking. If for some reason it doesn't understand something. Okay, so I'm going to show probably, you know, the interface from Plasma Minecraft, which is the nice thing here. This is a skill tab that currently doesn't work. That should show the skills that you have installed on the system. I mean, you have to think that this doesn't work. I think it's because of my packaging. It's not looking at the skills in the directory that they think they should be. But it's not necessarily really to make it work right now. Then you have a setting tab that you probably shouldn't touch. Because it allows the Plasma Minecraft widget to run and stop Minecraft in different... I'm not talking about him. It allows to stop and start Minecraft when you click on that button in different ways, depending on the installation way that you do it. But if you are using the Minecraft packages, then that's the only thing that you have to do there. And then this is interesting, is the skills tab which shows all the skills that are available and allows you to install them. As I said before, it's not currently the recognition of what the skills are installed and what the skills are not installed. It's not working. So some of these are already installed. And they're not recognized yet. But you can see here how it works. For example, let me search for something that I know that is not installed, the Hue skill that controls Philips Hue lights. Okay. You can click on Readme. Of course, it's a heavy GitHub user. So it goes directly to the GitHub page for the skill. And there you can see the Readme, which usually for, I think, all of the skills says how to install them, which basically is clicking on Install on that widget before. Some of them require a configuration. In general, it's not necessary to configure anything. And then it gives you the sentences that these skills understand. Okay. In this case, turn on my lights, set lights scene or something. In general, when you install a skill, as I said, my graph clones the repository from the skill into a skills directory. Okay. Oh, no. You can see that. That's better, right? Okay. So those are the skills that I have here. And as I said, the skills are cloned there. And then usually it uses PIP to install the dependencies of that skill. Since I remove all PIP usage from my graph, currently, if the skills you're installing is using some strange dependency that you don't have in your system, it just won't work. It doesn't do anything. Apart from logging in your logs that these skills need these requirements, these dependencies, so please be sure they install on your system. So that's what I said, that you should run Journal CTL and have a look at the user unit, my graph skills. Okay. I'm going to do that. Where you can see a lot of table information. As I said, this is mainly in development. If we go to the beginning, we can find some interesting things here. I think it's mostly time to end the talk, but I have to show something else yet. Hey, Minecraft. Remind me in four minutes to end the talk. You have a new reminder on three minutes and 59 seconds from now. To be sure that I give a couple of minutes at least for questions. And remove this. When the skills registers to the Microsoft skill demo, the skill can register a new vocabulary so that Microsoft recognizes more things. In this case, for example, we can see, let's do a better example. Where do I have this? I'm going to find it right now. I will show you just the example. Hey, Minecraft. Open Dolphin. So Dolphin is something that is not really, I mean, it doesn't exist in the skill, but the skill gets the list of applications that you have installed on your system and registers them as vocabulary. Then I can say, hey, Minecraft. Close Dolphin. I don't know what that means. Hey, Minecraft. Close Dolphin. Close. This is one of those things, by the way, that I said before that it's better not to look how it works. Because currently, the skills, when you ask it to close something, it just do an OS system, kill, all whatever. So, yeah. I hope it was a decision to do it quick so that it can improve later. I hope it's soon. What else do I want to show? I don't know if you know some nice thing about KDE that is the activities. You can have activities for different usage of your computer. So you can say, hey, Minecraft. Show activities. It shows the activities from the desktop. I don't have anything that probably, if I say, news and articles right now, it will just recognize it wrong and doesn't recognize that I'm referring to that. But in general, you can switch to another activity and control your computer with this. Something nice that I didn't show. Let's see. Let me run Kate. Hey, Minecraft. Type this is a test. Type in your request. You can use Minecraft to dictate things. And of course, if you speak English better than me, it will recognize you better than me. Let's see if I have something else that I want to show. This is nice. Hey, Minecraft, what's the weather like? Right now, it's scattered clouds and 21 degrees for a high of 24 and a low of 16. And if you look at, there are no windows. Cool. So you cannot check if it's right or not. Hey, Minecraft, will it rain today? Light rain is expected on Monday. So we can infer that. No. Hey, Minecraft, increase volume. There is a reminder to end the talk. Tell me to end reminder to stop the current reminder. Hey, Minecraft. Hey, Minecraft, stop the reminder. The reminder has been shut off. Okay. And what I was saying before. Hey, Minecraft, increase volume. And it was already at the max. But you see how it works. I don't know. Do you want to ask questions? Do you prefer to... I show you how our skill is developed quite quickly. Yeah. Wait, wait. Maybe I can get you here. This. How this works. Here. No. Okay. I have one question. Currently, there is only in English. Is it possible to locate two different languages? Yes, it's possible. And currently, English is the best supported language, of course. But if you go to... Did I close the console window? Yes. If you go to... Let me change this first. To Minecraft skills, for example... I don't know. Let me see the Wikipedia skill. Okay. I will take the opportunity to show a bit how the skills work. There is a set of directories here inside each of the skills. They define the vocabulary that the skill understands. Okay. So if you go inside this vocabulary, then there is a different directory for each of the languages. And you can go to English and you can see here that this skill understands when you mention Wikipedia, when you say, tell me about... So you can say, hey, Minecraft, tell me about London. Let me look up London. Okay. It says London is the capital and most popular citizen of the world. It says that for Wikipedia because the skills recognizer tells me about... The southeast of the island of Great Britain, London has been a major settlement for two years. And then you can also define other languages there. Right now, there is some basic support for German, for Italian, for French, I think. I think there was some Swedish support also. You have to think that the speech-to-text thing is done by Google. So it basically works in the language. And the text-to-speech is currently done with an application that the developers did, that is called Mimic. But you can configure it to also use Google to do that. So it should work. Okay. And the only thing that you have to do to support other languages is to implement all the vocabulary for all the skills. Not only for the recognition of the words, but also for the dialogue, for what Microsoft responds to the user, the Microsoft replies to the user. So in this case, for example, no entry phone, it has different sentences. That means there was no entry phone on Wikipedia about that subject. So that doesn't always always the same thing. But it just gets them randomly. And then there's also a regular expression that some skills have. That's a regular expression so that you can say different things and get which part of that sentence is the article that it should be looked for. Okay. And is there another question? Did I answer your question? Yeah, thank you. And I have also one question. When I asked to delete my disk, what did happen? That depends on the skills of the developer that developed the skill. Yeah. Of course, if you have a skill that recognizes delete my hard disk and you installed it, then yeah, it's partially your fault somehow. But yeah, in that case, you have to be careful with what you install, of course. Yeah. Which is something that probably should be thought about because right now the Microsoft developers mainly care about the machines that they have so that they don't care about, for example, installing credentials for the user on the home directory. And right now, as I said, they were shared on the shared directory. And if you have two users on the systems, well, they just accept changes and accept patches for related to the desktop. But most of the skills related to the desktop are done by Aditya Mera that I mentioned before. Yeah. Yeah. Installing skills is something, yeah. You have to be careful. And no questions? Any other questions? Yeah. Okay. First, thanks for all the work you're doing here because I know from my own experience how hard it can be to package such complex stuff. And even better, that you're on stream things because that makes it also easier. Now I'm also looking forward to integrate this or also other tools which are now emerging like the Susie project from Fossasia and other desktops and other distros. So thanks a lot. Thank you. In fact, I should say that I did the packaging, but Thomas Vatal and Thomas Bertol, I don't know if he's here, by the way, another developer, other developer languages Python maintainers also helped me a lot because they were, well, they had to review a lot of packages and they made me fix many things in those packages. So they also did a lot of work. Yeah. And before I finish, where's this? Let me say thanks. And I have a blog which currently has one entry that is related to Minecraft. And it explains how to install this on Open Susie at Amberweed. And I did the blog post two months ago so it doesn't work anymore because, finally, most of the dependencies were already in factory which they weren't on when I wrote that, but I plan to update it soon so you can see my second post there. Yeah. There's no other question. If there are more questions, you can ask me outside, probably. And thank you for coming.
open source virtual assistant. Something similar have to go through in order to submit source and a reply to the user. Okay. All this is done by a it's not using really, really the last version find some interesting things here. I think it's mostly time to end the talk, but I have to show skill can register a new vocabulary so that to look how it works. Because currently, the done with an application that the developers did, explains how to install this on Open Susie at