Lightning talk - Speech Recognition in the Browser
Jedryszek, Jakub
Mouse and keyboard are not the only ways how we can communicate with our devices. We already talk to our phones, gaming consoles, and some desktop applications. However, there are very few web applications that understand our speech. The bleeding edge versions of most popular web browsers are starting to support the most natural user input. Let’s take a look at Web Speech API, and learn how to add voice commands to web applications using pure JavaScript. Maybe it would be also worth to respond to users with voice using Web Synthesis API?
 I work on the Azure Management Portal. I have a session about how we built this one of the largest simple page applications in the world yesterday. So you can check out the video if you didn't have a chance to attend my session. You can find more information about me on my blog, JJ09.net. I encourage you to check it out and subscribe. And today I want to tell you about speech recognition in the browser, because probably many of you are using already a Siri on your iPhones or more legit Cortana on a Windows phone. But you might not realize that in Google Chrome, you can actually do speech recognition in the browser without any libraries. So I'll just open Google Chrome and I'll just open command line. Some error here, and that's okay. And I'll create new WebKit speech recognition object. And this object has only resolved callback function that I need to define. And this function has an event parameter. An event parameter has results property on it. And the results is an array of the results of the recognized speech. And I need to say transcript at the end. And then if I say start, hi, my name is Jacob. Oh, that's bad. Transcript. Thank you. Oh, yeah. Yeah, you're right. Transcript. All right. Thank you very much. And this print out my whatever I said. And you might notice that there's a two-dimensional array and you might wonder what is that and I'll get to that. But in addition to transcript, you also have confidence. So for example, you can say confidence here. And if I say something now, it says, oh, I don't know what you already said. My name is Jacob. It will give me like 95% of confidence that this is whatever I said. Another thing what I can do on a speech, WebKit speech recognition object, I have the property called interim results. And by default, it's false. So it's saying that it will only call this callback function when there is final result. But I can also see the intermediate results. So if I say interim results true, and if I say something now, it should print out whatever I'm saying at the time. So as you see, this is at some point in time, I don't only say and and then there are in other words and at the end, yeah, it sort of recognizes it, yeah, 89% of confidence. Let me switch it back to false. There's another thing called max alternatives. By default, it's one. So it's giving only one alternative, the best guess they do. But I can say max alternatives 10. And then I need to modify my callback a little bit because the alternative is actually this index of an array. So here I can say event. And because this is actually not a real array, so I need to do this trick for each and call and evt.results zero. And here I need to create the callback and say transcript. And I will print out confidence as well. And I hope I have enough brackets here. So now it should give me the alternatives. And you see I have different things what I said. I wasn't really clear. I guess none of these are right. NDC London rocks. Okay. So 78%. I said NDC London rocks. You could say NDT London or whatever. Was that? NDC? Is it by? Yeah. SR. Okay. So I will turn this off for now. I'll say one. And if you notice, you know, when I say something here, let me actually revert my callback to the original one. Okay. This one is good. So, you know, if I say something, for example, I start the sentence. Oops. I'm fine. What did I do wrong? Transcript. If I start the sentence. Okay. That's cool. But you could see that it's after I recognized something, it stopped listening. So there's also flag called continuous, which is set to false. If I set it to true, then this will add more to the... It will append the... Oops. What did I did? It will append the results to the array. And this is the first index. So I didn't want to do that. So the second index is for alternatives. First index is for the next recognized parts of whatever I set. So if I want to, for example, listen for the last thing that I said, I can say here, dbt.results.length. And I'll put the same in here. I hope it's true. And I start listening. Now I start my sentence. And then I can continue talking. And it will be still recognizing my speech until I say stop. So this is like another mode. Yeah. Sometimes it's funny. Probably there's some issue about it here. Let's hope for that. It's still experimental feature. So this is speech recognition. But what you can also do is you can emit voice so you can create new speech synthesis utterans. I will create new speech synthesis utterans. And I can say here, NDC London is great. And then I need to say speech synthesis, speak. And pass this message into here. Click enter. Do we have, do we have audio? Do we have audio? Okay. One more. NDC London is great. And you could notice some German accent here, right? So I can modify the message. There is a language. And you see the language is not set. For example, for sake of correctness, I will say it to British. NDC London is great. Much better, right? You can also do US accent. NDC London is great. And what is even cooler, you can also do Polish accent. Oops. And that's London is great. So you can play around with that. And you know, when I was playing mainly with speech recognition, because I was inspired in this idea that it would be so cool to add the voice commands to the website. And I put up the small library called voice commander. And you can find it on GitHub. It's just a wrapper on top of the WebKit speech recognition API. What it allows you to do, you can install it with NPM or Bower. You can just add a command to your code with simple JavaScript one minor. So you just say command is whatever I said, right? And the callback function is whatever you have to do when you add this command. I also put up the small website called bookslip. I hope you can see it. So if I say start, you see, it's asking me if I'm giving the permission to allow the microphone. If you're running the website on HTTP, it will ask you every time when you do reload. If you don't want to do that, you can run over HTTP. It will just ask for permission once. So here I would say allow. X favorites. Favorites. Top 10. Search JavaScript. So this is like an idea how you can put this into your website. And here I have, you know, on top, you could notice I hit the start and stop. And I also have get single command. So if I click get single command, it will just get one command and stop listening. So I say get single command. And it will ask me to allow again, fortunately. Home. Okay. Probably didn't recognize very well. Top 10. And go to command and stop listening. So please check out my voice commander library. You can find it on GitHub. Maybe you find it useful. And there are some links when you can see the demos on the website about WebSpeech API. There's even WebSpeech API spec. For now it's supported only by Google Chrome. If you want to have this in Microsoft Edge, you can do it. All you have to do is Microsoft Edge speech recognition API user voice, I think. Google for this. And there's user voice on the Windows website and you can see we have like 500 votes now. So if you add all of your votes, you can give maximum three votes per person. So we can like get another 100 out of this room, I think. And I think if we get closer, then Microsoft Edge team will be more encouraged to implement this feature in Microsoft Edge browser, too. So here is a link to my voice commander library on GitHub. And there's also this bookslip website I showed you. So thank you very much. And send me email if you have any questions about that. Thank you. Thank you.
 I work on the Azure Management Portal.  I have a session about how we built this one of the largest simple page applications in  the world yesterday.  So you can check out the video if you didn't have a chance to attend my session.  You can find more information about me on my blog, JJ09.net.  I encourage you to check it out and subscribe.  And today I want to tell you about speech recognition in the browser, because probably  many of you are using already a Siri on your iPhones or more legit Cortana on a Windows  phone.  But you might not realize that in Google Chrome, you can actually do speech recognition in  the browser without any libraries.  So I'll just open Google Chrome and I'll just open command line.  Some error here, and that's okay.  And I'll create new WebKit speech recognition object.  And this object has only resolved callback function that I need to define.  And this function has an event parameter.  An event parameter has results property on it.  And the results is an array of the results of the recognized speech.  And I need to say transcript at the end.  And then if I say start, hi, my name is Jacob.  Oh, that's bad.  Transcript.  Thank you.  Oh, yeah.  Yeah, you're right.  Transcript.  All right.  Thank you very much.  And this print out my whatever I said.  And you might notice that there's a two-dimensional array and you might wonder what is that and  I'll get to that.  But in addition to transcript, you also have confidence.  So for example, you can say confidence here.  And if I say something now, it says, oh, I don't know what you already said.  My name is Jacob.  It will give me like 95% of confidence that this is whatever I said.  Another thing what I can do on a speech, WebKit speech recognition object, I have the property  called interim results.  And by default, it's false.  So it's saying that it will only call this callback function when there is final result.  But I can also see the intermediate results.  So if I say interim results true, and if I say something now, it should print out whatever  I'm saying at the time.  So as you see, this is at some point in time, I don't only say and and then there are in  other words and at the end, yeah, it sort of recognizes it, yeah, 89% of confidence.  Let me switch it back to false.  There's another thing called max alternatives.  By default, it's one.  So it's giving only one alternative, the best guess they do.  But I can say max alternatives 10.  And then I need to modify my callback a little bit because the alternative is actually this  index of an array.  So here I can say event.  And because this is actually not a real array, so I need to do this trick for each and call  and evt.results zero.  And here I need to create the callback and say transcript.  And I will print out confidence as well.  And I hope I have enough brackets here.  So now it should give me the alternatives.  And you see I have different things what I said.  I wasn't really clear.  I guess none of these are right.  NDC London rocks.  Okay.  So 78%.  I said NDC London rocks.  You could say NDT London or whatever.  Was that?  NDC?  Is it by?  Yeah.  SR.  Okay.  So I will turn this off for now.  I'll say one.  And if you notice, you know, when I say something here, let me actually revert my callback to  the original one.  Okay.  This one is good.  So, you know, if I say something, for example, I start the sentence.  Oops.  I'm fine.  What did I do wrong?  Transcript.  If I start the sentence.  Okay.  That's cool.  But you could see that it's after I recognized something, it stopped listening.  So there's also flag called continuous, which is set to false.  If I set it to true, then this will add more to the...  It will append the...  Oops.  What did I did?  It will append the results to the array.  And this is the first index.  So I didn't want to do that.  So the second index is for alternatives.  First index is for the next recognized parts of whatever I set.  So if I want to, for example, listen for the last thing that I said, I can say here, dbt.results.length.  And I'll put the same in here.  I hope it's true.  And I start listening.  Now I start my sentence.  And then I can continue talking.  And it will be still recognizing my speech until I say stop.  So this is like another mode.  Yeah.  Sometimes it's funny.  Probably there's some issue about it here.  Let's hope for that.  It's still experimental feature.  So this is speech recognition.  But what you can also do is you can emit voice so you can create new speech synthesis  utterans.  I will create new speech synthesis utterans.  And I can say here, NDC London is great.  And then I need to say speech synthesis, speak.  And pass this message into here.  Click enter.  Do we have, do we have audio?  Do we have audio?  Okay.  One more.  NDC London is great.  And you could notice some German accent here, right?  So I can modify the message.  There is a language.  And you see the language is not set.  For example, for sake of correctness, I will say it to British.  NDC London is great.  Much better, right?  You can also do US accent.  NDC London is great.  And what is even cooler, you can also do Polish accent.  Oops.  And that's London is great.  So you can play around with that.  And you know, when I was playing mainly with speech recognition, because I was inspired  in this idea that it would be so cool to add the voice commands to the website.  And I put up the small library called voice commander.  And you can find it on GitHub.  It's just a wrapper on top of the WebKit speech recognition API.  What it allows you to do, you can install it with NPM or Bower.  You can just add a command to your code with simple JavaScript one minor.  So you just say command is whatever I said, right?  And the callback function is whatever you have to do when you add this command.  I also put up the small website called bookslip.  I hope you can see it.  So if I say start, you see, it's asking me if I'm giving the permission to allow the  microphone.  If you're running the website on HTTP, it will ask you every time when you do reload.  If you don't want to do that, you can run over HTTP.  It will just ask for permission once.  So here I would say allow.  X favorites.  Favorites.  Top 10.  Search JavaScript.  So this is like an idea how you can put this into your website.  And here I have, you know, on top, you could notice I hit the start and stop.  And I also have get single command.  So if I click get single command, it will just get one command and stop listening.  So I say get single command.  And it will ask me to allow again, fortunately.  Home.  Okay.  Probably didn't recognize very well.  Top 10.  And go to command and stop listening.  So please check out my voice commander library.  You can find it on GitHub.  Maybe you find it useful.  And there are some links when you can see the demos on the website about WebSpeech API.  There's even WebSpeech API spec.  For now it's supported only by Google Chrome.  If you want to have this in Microsoft Edge, you can do it.  All you have to do is Microsoft Edge speech recognition API user voice, I think.  Google for this.  And there's user voice on the Windows website and you can see we have like 500 votes now.  So if you add all of your votes, you can give maximum three votes per person.  So we can like get another 100 out of this room, I think.  And I think if we get closer, then Microsoft Edge team will be more encouraged to implement  this feature in Microsoft Edge browser, too.  So here is a link to my voice commander library on GitHub.  And there's also this bookslip website I showed you.  So thank you very much.  And send me email if you have any questions about that.  Thank you.  Thank you.
Lightning_talk___Speech_Recognition_in_the_Browser_-_Jakub_Jedryszek_0_275.png Lightning_talk___Speech_Recognition_in_the_Browser_-_Jakub_Jedryszek_276_611.png Lightning_talk___Speech_Recognition_in_the_Browser_-_Jakub_Jedryszek_612_791.png Lightning_talk___Speech_Recognition_in_the_Browser_-_Jakub_Jedryszek_792_1043.png Lightning_talk___Speech_Recognition_in_the_Browser_-_Jakub_Jedryszek_1044_1151.png Lightning_talk___Speech_Recognition_in_the_Browser_-_Jakub_Jedryszek_1152_1535.png Lightning_talk___Speech_Recognition_in_the_Browser_-_Jakub_Jedryszek_1536_4547.png Lightning_talk___Speech_Recognition_in_the_Browser_-_Jakub_Jedryszek_4548_14219.png Lightning_talk___Speech_Recognition_in_the_Browser_-_Jakub_Jedryszek_14220_14675.png Lightning_talk___Speech_Recognition_in_the_Browser_-_Jakub_Jedryszek_14676_14879.png Lightning_talk___Speech_Recognition_in_the_Browser_-_Jakub_Jedryszek_14880_15851.png Lightning_talk___Speech_Recognition_in_the_Browser_-_Jakub_Jedryszek_15852_16223.png Lightning_talk___Speech_Recognition_in_the_Browser_-_Jakub_Jedryszek_16224_16319.png Lightning_talk___Speech_Recognition_in_the_Browser_-_Jakub_Jedryszek_16320_17051.png Lightning_talk___Speech_Recognition_in_the_Browser_-_Jakub_Jedryszek_17052_17279.png Lightning_talk___Speech_Recognition_in_the_Browser_-_Jakub_Jedryszek_17280_17759.png Lightning_talk___Speech_Recognition_in_the_Browser_-_Jakub_Jedryszek_17760_19079.png Lightning_talk___Speech_Recognition_in_the_Browser_-_Jakub_Jedryszek_19080_19392.png
I work on the Azure Management Portal. I have a session about how we built this one of the largest simple page applications in the world yesterday. So you can check out the video if you didn't have a chance to attend my session. You can find more information about me on my blog, JJ09.net. I encourage you to check it out and subscribe. And today I want to tell you about speech recognition in the browser, because probably many of you are using already a Siri on your iPhones or more legit Cortana on a Windows phone. But you might not realize that in Google Chrome, you can actually do speech recognition in the browser without any libraries. So I'll just open Google Chrome and I'll just open command line. Some error here, and that's okay. And I'll create new WebKit speech recognition object. And this object has only resolved callback function that I need to define. And this function has an event parameter. An event parameter has results property on it. And the results is an array of the results of the recognized speech. And I need to say transcript at the end. And then if I say start, hi, my name is Jacob. Oh, that's bad. Transcript. Thank you. Oh, yeah. Yeah, you're right. Transcript. All right. Thank you very much. And this print out my whatever I said. And you might notice that there's a two-dimensional array and you might wonder what is that and I'll get to that. But in addition to transcript, you also have confidence. So for example, you can say confidence here. And if I say something now, it says, oh, I don't know what you already said. My name is Jacob. It will give me like 95% of confidence that this is whatever I said. Another thing what I can do on a speech, WebKit speech recognition object, I have the property called interim results. And by default, it's false. So it's saying that it will only call this callback function when there is final result. But I can also see the intermediate results. So if I say interim results true, and if I say something now, it should print out whatever I'm saying at the time. So as you see, this is at some point in time, I don't only say and and then there are in other words and at the end, yeah, it sort of recognizes it, yeah, 89% of confidence. Let me switch it back to false. There's another thing called max alternatives. By default, it's one. So it's giving only one alternative, the best guess they do. But I can say max alternatives 10. And then I need to modify my callback a little bit because the alternative is actually this index of an array. So here I can say event. And because this is actually not a real array, so I need to do this trick for each and call and evt.results zero. And here I need to create the callback and say transcript. And I will print out confidence as well. And I hope I have enough brackets here. So now it should give me the alternatives. And you see I have different things what I said. I wasn't really clear. I guess none of these are right. NDC London rocks. Okay. So 78%. I said NDC London rocks. You could say NDT London or whatever. Was that? NDC? Is it by? Yeah. SR. Okay. So I will turn this off for now. I'll say one. And if you notice, you know, when I say something here, let me actually revert my callback to the original one. Okay. This one is good. So, you know, if I say something, for example, I start the sentence. Oops. I'm fine. What did I do wrong? Transcript. If I start the sentence. Okay. That's cool. But you could see that it's after I recognized something, it stopped listening. So there's also flag called continuous, which is set to false. If I set it to true, then this will add more to the... It will append the... Oops. What did I did? It will append the results to the array. And this is the first index. So I didn't want to do that. So the second index is for alternatives. First index is for the next recognized parts of whatever I set. So if I want to, for example, listen for the last thing that I said, I can say here, dbt.results.length. And I'll put the same in here. I hope it's true. And I start listening. Now I start my sentence. And then I can continue talking. And it will be still recognizing my speech until I say stop. So this is like another mode. Yeah. Sometimes it's funny. Probably there's some issue about it here. Let's hope for that. It's still experimental feature. So this is speech recognition. But what you can also do is you can emit voice so you can create new speech synthesis utterans. I will create new speech synthesis utterans. And I can say here, NDC London is great. And then I need to say speech synthesis, speak. And pass this message into here. Click enter. Do we have, do we have audio? Do we have audio? Okay. One more. NDC London is great. And you could notice some German accent here, right? So I can modify the message. There is a language. And you see the language is not set. For example, for sake of correctness, I will say it to British. NDC London is great. Much better, right? You can also do US accent. NDC London is great. And what is even cooler, you can also do Polish accent. Oops. And that's London is great. So you can play around with that. And you know, when I was playing mainly with speech recognition, because I was inspired in this idea that it would be so cool to add the voice commands to the website. And I put up the small library called voice commander. And you can find it on GitHub. It's just a wrapper on top of the WebKit speech recognition API. What it allows you to do, you can install it with NPM or Bower. You can just add a command to your code with simple JavaScript one minor. So you just say command is whatever I said, right? And the callback function is whatever you have to do when you add this command. I also put up the small website called bookslip. I hope you can see it. So if I say start, you see, it's asking me if I'm giving the permission to allow the microphone. If you're running the website on HTTP, it will ask you every time when you do reload. If you don't want to do that, you can run over HTTP. It will just ask for permission once. So here I would say allow. X favorites. Favorites. Top 10. Search JavaScript. So this is like an idea how you can put this into your website. And here I have, you know, on top, you could notice I hit the start and stop. And I also have get single command. So if I click get single command, it will just get one command and stop listening. So I say get single command. And it will ask me to allow again, fortunately. Home. Okay. Probably didn't recognize very well. Top 10. And go to command and stop listening. So please check out my voice commander library. You can find it on GitHub. Maybe you find it useful. And there are some links when you can see the demos on the website about WebSpeech API. There's even WebSpeech API spec. For now it's supported only by Google Chrome. If you want to have this in Microsoft Edge, you can do it. All you have to do is Microsoft Edge speech recognition API user voice, I think. Google for this. And there's user voice on the Windows website and you can see we have like 500 votes now. So if you add all of your votes, you can give maximum three votes per person. So we can like get another 100 out of this room, I think. And I think if we get closer, then Microsoft Edge team will be more encouraged to implement this feature in Microsoft Edge browser, too. So here is a link to my voice commander library on GitHub. And there's also this bookslip website I showed you. So thank you very much. And send me email if you have any questions about that. Thank you. Thank you.
probably many of you are using already a Siri on is a language. And you see the language is not that it would be so cool to add the voice commands to the website. And I put up the small command. I also put up the small website called JavaScript. So this is like an idea how you can Maybe you find it useful. And there are some Microsoft Edge speech recognition API user voice, on the Windows website and you can see we have