Accessible input for readers, coders, and hackers
Williams-King, David
When people develop carpal tunnel or various medical conditions, it can be difficult to use mainstream input mechanisms like keyboards, mice, and phone touchscreens. Ideally, accessible input mechanisms can be added to mainstream computers and phones. I will give two example demos. The first is using voice or eyelid blinks to control an ebook reader on a standard Android phone. The second is using speech recognition to interact with a Linux desktop, even to perform complicated tasks such as programming. When people develop carpal tunnel or various medical conditions, it can be difficult to use mainstream input mechanisms like keyboards, mice, and phone touchscreens. Such users might have to rely on speech recognition, eye tracking, head gestures, etc. Technology allows devices to understand these mechanisms as user input, and hence provide agency and a communication channel to the rest of the world. Every person's case is a bit different. Sometimes, a fully custom system has to be designed, such as for Stephen Hawking. A team from Intel worked through many prototypes with him, saying: "The design [hinged] on Stephen. We had to point a laser to study one individual." With custom hardware and custom software to detect muscle movements in his cheek, he still could only communicate a handful of words per minute. In less severe situations, the goal is often to add accessible input mechanisms to mainstream computers and phones. Similarly, a blind person will adapt to using a normal smartphone, despite not being able to see the screen. It is not economical to design myriad variants of hardware to handle many different users who all have slightly different needs. Adapting mainstream devices allows a wide range of existing software to be used, from email clients to Google maps, without reinventing everything. In my own case, I have a medical condition and cannot use my hands more than a certain threshold in a day. In order to keep reading books, I designed a speech recognition system and also an eyelid blink system to control an ebook reader app. As a computer programmer, I used to make heavy use of a keyboard to write code. With modern speech recognition, it is possible to design a spoken language that allows precise control over a computer. I will demo an open source voice coding system, and describe how it can be adapted to do any task on a desktop computer.
 Alright, so again, let's introduce the next talk, Accessible Inputs for Readers, Coders and Hackers, the talk by David Williams King about custom, well not off the shelf, but custom accessibility solutions. He will give you some demonstrations and that includes his own custom made voice input and eyelid blink system. Here is David Williams King. Thank you for the introduction. Let's go ahead and get started. So yeah, I'm talking about accessibility, particularly accessible input for readers, coders and hackers. So what do I mean by accessibility? I mean people that have physical or motor impairments, this could be due to repetitive strain injury, carpal tunnel, all kinds of medical conditions. If you have this type of thing, you probably can't use a normal computer keyboard, computer mouse or even a phone touch screen. However, technology does allow users to interact with these devices just using different forms of input. And it's really valuable to these people because being able to interact with a device provides some agency, they can do things on their own and it provides a means of communication with the outside world. So it's an important problem to look at. And it's one I care about a lot. Let's talk a bit about me for a moment. I'm a system security person. I did a PhD in cybersecurity at Columbia. If you're interested in low level software defenses, you can look that up. And I'm currently the CTO at a startup called Alpha Secure. I started developing medical issues in around 2014. And as a result of that, in an ongoing fashion, I can only type a few thousand keystrokes per day. Roughly 15,000 is my maximum. That sounds like a lot, but imagine you're typing at 100 words per minute. That's 500 characters per minute, which means it takes you 30 minutes to hit 15,000 characters. So essentially, I can work like the equivalent of a fast programmer for half an hour. And then after that, I would be unable to use my hands for anything, including like, you know, preparing food for myself or opening and closing doors and so on. So I have to be very careful about my hand use and actually have a little program that you can see on the slide there that measures the keystrokes for me so I can tell when I'm going over. So what do I do? Well, I do a lot of pair programming for sure. I log into the same machine as other people and we work together. I'm also a very heavy user of speech recognition. And I give a talk at that about voice coding with speech recognition at the Hope 11 conference. So you can go check that out if you're interested. So when I talk about accessible input, I mean different ways that a human can provide input to a computer. So ergonomic keyboards are a simple one. Speech recognition, eye tracking or gaze tracking. So you can see where you're looking or where you're pointing your head and maybe use that to replace a mouse. That's head gestures, I suppose. And there's always this distinction between bespoke, like custom input mechanisms and somewhat mainstream ones. So I'll give you some examples. You've probably heard of Stephen Hawking. He's a very famous professor and he was actually a bit of an extreme case. He was diagnosed with ALS when he was 21. So his physical abilities degraded over the years because he lived for many decades after that. And he went through many communication mechanisms. Initially, his speech changed so that it was only intelligible to his family and close friends, but he was still able to speak. And then after that, he would work with a human interpreter and raise his eyebrows to pick various letters. And then, keep in mind, this is like the 60s or 70s, right? So computers were not really where they are today. Later, he would operate a switch with one hand, just like on off, on off, kind of Morse code and select from a bank of words. And that was around 15 words per minute. Eventually, he was unable to move his hand. So a team of engineers from Intel worked with him and they figured out they were trying to do like brain scans and all kinds of stuff. But again, this was like in the 80s. So there was not, not too much they could do. So they basically just created some custom software to detect muscle movements in his cheek. And he used that with predictive, predictive words the same way that a phone, smartphone keyboard will predict like which word you want to say next. Stephen Hawking used something similar to that except instead of swiping on a phone, he was moving his, his cheek muscles. So that's obviously a sequence of like highly customized input mechanisms for, for someone and very, very specialized for that person. I also want to talk about someone else named Professor Song Muk Lee, whom I've met that was me when I had more of a beard than I do now. He, he's a professor at Seoul National University in South Korea. And he, he's sometimes called like the Korean Stephen Hawking because he's a big advocate for people with disabilities and whatnot. Anyway, what he uses is you can see a little orange device near his mouth there. He, it's called a sip and puff mouse. So he can blow into it and suck air through it and also move it around. And that acts as a mouse cursor on the Android device in front of him. It will move the cursor around and click when he, when he blows air and so on. So that combined with speech recognition, lets him use mainstream Android hardware. He still has access to, you know, email apps and like web browsers and like maps and everything that comes on a normal Android device. So he's way more capable than Stephen Hawking was because Stephen Hawking could communicate, but just to a person at a very slow rate, right? Part of it is due to the nature of his injury, but it's also a testament to how far the technology has improved. So let's talk a little bit about what makes good accessibility. I think performance is very important, right? You want high accuracy, you don't want typos, low latency, I don't want to speak and then five seconds later have words appear. It's too, it's too long, especially if you have to make corrections, right? And you want high throughput, which we already talked about. Oh, I forgot to mention Stephen Hawking had like, you know, 15 words per minute. A normal person speaking is 150. So that's a big difference. The higher throughput you can get the better. And for input accessibility, I think, and this is not scientific, this is just what I've learned from using it myself and observing many of these systems. I think it's important to get completeness, consistency and customization. For completeness, I mean, can I do any action? So Stephen or Professor Song-Luk Lee, his orange mouth input device, the Sip and Puff, is quite powerful, but it doesn't let him do every action. For example, for some reason, when he gets an incoming call, the input doesn't work. So he has to call over a person physically to like tap the accept call button or the reject call button, which is really annoying, right? If you don't have completeness, you can't be fully independent. Consistency, very important as well. The same way we develop motor memory for a muscle memory for a keyboard, you develop memory for any types of patterns that you do. But if the thing you say or the thing you do keeps changing in order to do the same action, that's not good. And finally, customization. So the learning curve for beginners is important for any accessibility device, but designing for expert use is almost more important because anyone who uses an accessibility interface becomes an expert at it. The example I like to give is screen readers, like a blind person using a screen reader on a phone, they will crank up the speed at which the speech is being produced. And I actually met someone who made his speech 16 times faster than normal human speech. I could not understand it at all. But he could understand it perfectly. And that's just because he used it so much that he's become an expert at its use. Let's analyze ergonomic keyboards just for a moment because it's fun. They are kind of like a normal keyboard. They'll have a slow pace when you're starting to learn them. But once you're good at it, you have very good accuracy, like instantaneous low latency, right? You press the key, the computer receives it immediately. And very high throughput, as high as you are on a regular keyboard. So they're actually fantastic accessibility devices, right? They're completely compatible with original keyboards. And if all you need is an ergonomic keyboard, then you're in luck because it's a very good accessibility device. I'm going to talk about two things, computers, but also Android devices. So let's start with Android devices. Yes, the built-in voice recognition in Android is really incredible. So even though the microphones on the devices aren't great, Google has just collected so much data from so many different sources that they've built, like, better than human accuracy for their voice recognition. The voice accessibility interface is kind of so-so. We'll talk about that in a bit. That's the interface where you can control the Android device entirely by voice. For other input mechanisms, you could use like a Sip and Puff device, or you could use physical styluses. That's something that I do a lot actually, because for me, my fingers get sore. And if I can hold a stylus in my hand and kind of not use my fingers, then that's, you know, very effective. So, and the Elacom styluses from a Japanese company are the lightest I've found, and they don't require a lot of force. So the ones at the top there are, they're like 12 grams, and the one at the bottom is 4.7 grams. And you've required almost no force to use them, so very nice. On the left there, you can see the Android speech recognition is built into the keyboard now, right? You can just press that and start speaking in sports different languages, and it's very accurate. It's very nice. And actually, when I was working at Google for a bit, I talked to the speech recognition team, and I was like, why are you doing on-server speech recognition? You should do it on the devices. But of course, Android devices are, they're all very different, and many of them are not very powerful, so they were having trouble getting satisfactory speech recognition on the device. So for a long time, there's some server latency, server lag, right? You do speech recognition and you wait a bit. And then sometime this year, I just was using speech recognition, and it became so much faster. I was extremely excited, and I looked into it, and yeah, they just switched on my device at least, they switched on the on-device speech recognition model. And so now it's incredibly fast and also incredibly accurate. I'm a huge fan of it. On the right-hand side, we can actually see the voice access interface. So this is meant to allow you to use a phone entirely by voice. Again, while I was at Google, I tried the beta version before it was publicly released, and I was like, this is pretty bad. Mostly because it lacked completeness. There would be things on the screen that would not be selected. So here we see show labels, and then I can say like four, five, six, whatever, to tap on that thing. But as you can see at the bottom there, there's like a Twitter web app link, and there's no number on it. So if I want to click on that, I'm out of luck. And this is actually a problem in the design of the accessibility interface. It doesn't expose the full DOM. It exposes only a subset of it, and so an accessibility mechanism can't ever see those other things. And furthermore, the way the Google speech recognition works, they have to reestablish a new connection every 30 seconds. And if you're in the middle of speaking, it will just throw away whatever you were saying because it just decided it had to reconnect, which is really unfortunate. They later released that publicly, and then sometime this year they did an update, which is pretty nice. It now has like a mouse grid, which solves a lot of the completeness problems. You can use a grid to narrow down somewhere on the screen and then tap there. But the server issues and the expert use is still not good. Like, okay, if I want to do something with a mouse grid, I have to say mouse grid on, six, five, mouse grid off. And I can't combine those together. So there's a lot of latency. And it's not really that fun to use, but better than nothing, absolutely. I just want to really briefly show you as well that this same feature of like being able to select links on a screen is available on desktops. This is a plugin for Chrome called Vimium. And it's very powerful because you can then combine this with keyboards or other input mechanisms. And this one is complete. It uses the entire DOM and anything you can click on will be highlighted. So very nice. I just want to give a quick example of me using some of these systems. So I've been trying to learn Japanese. And there's a couple of highly regarded websites for this, but they're not consistent when I use the browser show labels. Like, you know, the thing to press next page or something like that, or like, you know, I give up or whatever is, it keeps changing. So the letters that are being used keep changing. And that's because of the dynamic way that they're generating the HTML. So not really very useful. What I do instead is I use a program called Anki. And that has very simple shortcuts in his desktop app, one, two, three, four. So it's nice to use and consistent. And it syncs with an Android app. And then I can use my stylus on the Android device. So it works pretty well. But even so, you know, as you can see from the chart in the bottom there, there are many days when I can't use this, even though I would like to, because I've overused my hands or overused my voice. When I'm using voice recognition all day, every day, I do tend to lose my voice. And as you can see from the graph, sometimes I lose it for like a week or two at a time. So same thing with any accessibility interface, you know, you got to use many different techniques. And it's always, it's never perfect. It's just the best you can do at that moment. Something else I like to do is read books. I read a lot of books. And I love eBook readers, you know, the dedicated eInk displays, you can read them in sunlight, they last forever battery-wise. Unfortunately, it's hard to add like other input mechanisms to them. They don't have microphones or other sensors. And you can't really install custom software on them. But for Android-based devices, and they're also like eBook reading apps for Android devices, they have everything. You can install custom software and they have microphones and many other sensors. So I made two apps that allow you to read eBooks with an eBook reader. The first one is Voice Next Page. It's based on one of my speech recognition engine called Silvius. And it does do server-based recognition. So you have to capture all the audio, use 300 kilobits a second to send it to the server and recognize things like Next Page, Previous Page. However, it doesn't cut out every 30 seconds. It keeps going. So that's one win for it, I guess. And it is published in the Play Store. Huge thanks to Sarah Leventhal who did a lot of the implementation, very complicated to make an accessibility app on Android. But we persevered and works quite nicely. So I'm going to actually show you an example of Voice Next Page. This is a... Over here, this is my phone on the left-hand side. Just captured so that you guys can see it. So here's the Voice Next Page. And basically, the connection is green. I can do the servers up and running and so on. I just press Start and then I'll switch to an Android reading app and say Next Page, Previous Page. I won't speak otherwise because it will chapel everything I'm saying. Next Page, Next Page, Previous Page. Center, Center, Foreground. Stop listening. So that's a demo of the Voice Next Page. And it's extremely helpful. I built it a couple of years ago along with Sarah. And I use it a lot. So yeah, you can go ahead and download it if you guys want to try it out. And the other one is called Blink Next Page. So the idea for this... I got this idea from a research paper this year that was studying islet gestures. I didn't use any of their code, but it's a great idea. So the way this works is you detect blinks by using the Android camera and then you can trigger an action like turning pages in an eBook reader. This actually doesn't need any networking. It's able to use the on device face recognition models from Google. And it is still under development, so it's not on the play story yet. But it is working. And please contact me if you want to try it. So just give me one moment to set that demo up here. And so I'm going to use... The main problem with this... The main problem with this current implementation is that it uses two devices. So that was easier to implement. And I use two devices anyway, but obviously I want a one device version if I'm actually going to use it for anything. So here's how this works. This device I point at me at my eyes. The other device I put wherever it's convenient to read. Oops, sorry. And if I blink my eyes, the phone will buzz once it detects that I blink my eyes and it will turn the page automatically on the other Android device. Now I have to blink both my eyes for half a second. If I want to go backwards, I can blink just my left eye. And if I want to go forwards, quickly, I can blink my right eye and hold it. Anyway, it does have some false positives. That's why I like you can go backwards. In case it detects that you've accidentally flipped the page. And lighting is also very important. Like if I have a light behind me, then this is not going to be able to identify whether my eyes are open or closed properly. So it has some limitations, but very simple to use. So I'm a big fan. Okay, so that's enough about Android devices. Let's talk very briefly about desktop computers. So if you're going to use a desktop computer, of course, try using that show labels plugin in a browser for native apps, you can try Dragon Naturally Speaking, which is fine if you're just using basic things. But if you're trying to do complicated things, you should definitely use a voice coding system. You could also consider using eye tracking to replace a mouse. I personally, I don't use that. I find it hurts my eyes, but I do use a track ball with very little force and a walk on tablet. Some people will even scroll up and down by humming, for example, but I don't have that set up. There's a bunch of nice talks out there on voice coding. The top left is Tavis Reds talk from many years ago that got many of us interested. Emily Shia gave a talk there about like best practices for voice coding. And then I gave a talk a couple years ago at the Hope 11 conference, which you can also check out. It's mostly out of date by now, but it's still interesting. So there are a lot of voice coding systems. The sort of grandfather of the mall is Dragonfly. It's become a grammar standard. Caster is if you're willing to memorize lots of unusual words, you can become much better, much faster than I currently am at voice coding. A Nia is how you originally used Dragon to work on a Linux machine, for example, because Dragon only runs on Windows. Talon is a closed source program, which is, but it's very, very powerful, has a big user base, especially for Mac OS. There are ports now. And Talon used to use Dragon, but it's now using a speech system from Facebook. Sylveus is the system that I created. The models are not very accurate, but it's a nice architecture where there's client server, so it makes it easy to build things like the voice next page. So the voice next page was using Sylveus. And then the most recent one I think on this list is Calde Active Grammar, which is extremely powerful and extremely customizable, and it's also open source. It works on all platforms, so I really highly recommend that. So let's talk a bit more about Calde Active Grammar. But first, for voice coding, I've already mentioned you have to be careful how you use your voice, right? Breathe through your belly. Don't tighten your muscles and breathe from your chest. Try to speak normally. And I'm not particularly good at this. Like, you'll hear me when I'm speaking commands. My inflection changes, so I do tend to overuse my voice. But yeah, I just have to be conscious of that. The microphone hardware does matter. I do recommend like a blue Yeti on a microphone arm that you can pull and put close to your face like this. I'll use this one for my speaking demo. And yeah, and the other thing is your grammar is fully customizable. So if you keep saying a word and the system doesn't recognize it, just change it to another word. And it's complete in the sense you can type any key on the keyboard. And the most important thing for expert use or customizability is that you can do chaining. So with a voice coding system, you can say multiple commands at once. And it's a huge time savings. You'll see what I mean when I give a quick demo. When I do voice coding, I'm a very heavy Vim and Tmux user. You know, there have been, I've worked with many people before, so I have some cheat sheet information there. So if you're interested, you can go check that out. But yeah, let's just do a quick demo of voice coding here. Turn this mic on. Desk left to control delta, open new terminal, Charlie, Delta space slash Tango mic papa enter command Vim, hotel, hotel point, Charlie, papa, papa, enter. India hash word include space lango. India Oscar word stream wrangle, enter, enter. Indian noitango space word main. Nope, my car to Indian noi. Landrun space lace enter, enter race up tab. Word print Fox scratch nope. Code standard Charlie Oscar uniform Tango space lango lango space quote sentence Hello voice coding bang. Scratch six Delta India, noi golf bang back slash noi quote. semi colon act, Sky Fox mic Romeo noi Oscar word return space number zero semi colon act. Vim save and quit. Golf plus plus space hotel hotel tab minus Oscar space hotel hotel. Enter point slash hotel hotel enter. Desk right to. So that's just a quick example of voice coding. You can use it to write any programming language. You can use it to control anything on your desktop. It's very powerful. It has a bit of a learning curve, but it's very powerful. So the creator of Calde active grammar is also named David. I'm named David, but just a coincidence. And he says of Calde active grammar that I haven't typed with a keyboard in many years and Calde active grammar is bootstrapped in that I have been developing it entirely using the previous versions of it. So David has a medical condition that means he has very low dexterity. So it's hard for him to use a keyboard. And yeah, he basically got Calde active grammar working through the skin of his teeth or something and then continues to develop it using it. And yeah, I'm a huge fan of the project. I haven't contributed much, but I did give some of the harbor resources like GPU and CPU compute compute resources to allow training to happen. But I would also like to show you a video of David using Calde active grammar just so you can see it as well. So the other thing about David is that he has he has a speech impediment or a speech, I don't know, an accent or whatever. So it's difficult for a normal speech recognition system to understand him. And you might have trouble understanding him here, but you can see in the lower right what the speech system understands that he's saying. Oh, I realized that I do need to switch something in OBS so that you guys can hear it. Sorry. There we go. Tim. Number one. Similar comments space. Number one, one, one. Content line. Content, script, enter. Text. Timer assignment. Aposite. Content, enter two times. And then, past rent space. And then, Anyway, you get the idea. And hopefully you guys are able to hear that. If not, you can also find this on the website that I'm going to show you at the end. Oh, one other thing I want to show you about this is David has actually set up this humming to scroll, which I think is pretty cool. Of course, I got and turned off the OBS there, but he's just going, and it's understanding that and scrolling down. So something that I'm able to do with my track ball, but that he's using his voice for. So pretty cool. So I'm almost done here in summary. Good input accessibility means you need completeness, consistency, and customization. You need to be able to do any action that you could do with the other input mechanisms. And doing the same input should have the same action. And remember, your users will become experts. So the system needs to be designed for that. For ebook reading. Yes, we're trying. I'm trying to allow anyone to read, even if they're experiencing some severe physical or motor impairment, because I think that gives you a lot of power to be able to turn the pages and read your favorite books. And for speech recognition, yeah, Android speech recognition is very good. Sylvia's accuracy is not so good, but it's easy to use quickly for experimentation and to make other types of things like voice next page. And please do check out Caldea Active Grammar if you have some serious need for voice recognition. Lastly, I put all of this onto a website, voxhub.io. So you can see voice next page, link next page, Caldea Active Grammar, and so on, just instructions for how to use it and how to set it up. So please do check that out. And tons of acknowledgments, lots of people that have helped me along the way, but I want to especially call out Professor Sang-Muk Lee, who actually invited me to Korea a couple of times to give talks, a big inspiration. And of course, David Zuro has actually been able to bootstrap into a fully voice coding environment. So that's all I have for today. Thank you very much. Right. I suppose I'm back on the air. So let me see. I want to remind everyone before we go into the Q&A that you can ask your questions for this talk on IRC. The link is under the video, or you can use Twitter, or the FETIverse with the hashtag RC32. Again, I'll hold it up here, RC number three, TWO. And wow, thanks for talking, David. That was really interesting. Thanks for talking, David. I think we have a couple of questions from the Signal Angels. Before that, I just want to say I recently spent some time playing with the voiceover system in iOS, and that can now actually tell you what is on a photo, which is kind of amazing. Oh, by the way, I can't hear you here on the mobile. Yeah, sorry. I wasn't saying anything. Yeah, no, it's so I focus mostly on input accessibility, right, which is like, how do you get data to the computer? But there's been huge improvements in the other way around as well, right, the computer? Yeah, so we have about, let's see, five, six minutes left, at least for Q&A. We have a question by Toby plus plus, he asks, your next page application looks cool. Do you have statistics of how many people use it or found it on the app store? Not very many. The voice next page was advertised only so far as a little academic poster. So I've gotten a few people to use it. But I run eight concurrent workers. And we've never hit more than that. So not super, not super popular, but I do hope that some people will see it because of this talk and go and check it out. Cool. Next question, how error prone are the speech recognition systems at all? E.g. Can you do coding while doing workouts? So one thing about speech recognition is very sensitive to the microphone. So when you're doing it, you can't do it. Any mistakes, right? That's the thing about having low latency, you just say something, and you watch it, and you make sure that it was what you wanted to say. I don't know exactly how many words per second, words per minute, I can say with voice coding, but I can say it much faster than regular speech. So I would say at least like 200, maybe 300 words per minute. So it's actually a very high bandwidth. Question from Pepe, JN, Divos. Any advice for software authors to make their stuff more accessible? There are good web accessibility guidelines. So if you're just making a website or something, I would definitely follow those. They tend to be focused more on people that are blind because that is, you know, it's more of an obvious fail. Like they just can't interact at all with your website. But things like, you know, if Duolingo, for example, had used the same accessibility access tag on their like next button, then they would always be the same letter for me. And I wouldn't have to be like, Fox, Charlie, Fox, Delta, Fox, something changes all the time. So I think consistency is very important. And integrating with any existing accessibility API is also very important, web APIs, Android APIs, and so on. Because, you know, we can't make every program out there like voice compatible, we just have to meet in the middle where they interact at the keyboard layer or the all American has a question. I wonder if these systems use similar approaches like stenography with mnemonics, or if there's any projects working having that in mind? A very good question. So the first thing everyone uses is the NATO phonetic alphabet to spell letters, for example, alpha Bravo, Charlie. Some people then will substitute letters for things that are too long, like November, I use no way. Sometimes the speech system doesn't understand you. Whenever I said alpha, Dragon was like, Oh, you're saying offer. So I had changed it. It's arch for me arch bravchar. So and also most of these grammars are in a common grammar format. They are written in Python, and they're compatible with dragonfly. So you can grab a grammar for, I don't know, for a Nia and get it to work with Caldeactive grammar with very little effort. I actually have a grammar that works on both a Nia and Caldeactive grammar. And that's what I use. So there's a bit of lingua franca, I guess you can kind of guess what other people are using. But at the same time, there's a lot of customization, you know, because people change words, they add their own commands, they change words based on what the speech system understands. LEB asks, is there an online community you can propose for accessibility technologies? There's a there's an amazing forum for anything related to voice coding. All the developers of new voice coding software are there. Sorry, I just need to drink. So it's a really fantastic resource. I do link to it from voxhub.io. I believe is at the bottom of the Caldeactive grammar page. So you can definitely check that out. For general accessibility, I don't know, I could recommend the accessibility mailing list at Google, but that's only if you work at Google. Other than that. Yeah, I think it depends on your community, right? I think if you're looking for web accessibility, you could go for some Mozilla mailing lists and so on. If you're looking for desktop accessibility, then maybe you could go find some stuff about the Windows speech API and the Windows accessibility. One last question from Joe Nielsen. Could there be legal issues if you make an ebook into audio? I'm not sure what that refers to. Yeah, so if you are like doing, if you're using a screen reader and you like, you know, try to get it to read out the contents of a of an ebook, right? So most of the time there are fair use exceptions for copyright law, even in the US. And making a copy of yourself for personal purposes so that you can access it is usually considered fair use. If you were trying to commercialize it or make money off of that or like, I don't know, you're a famous streamer and all you do is highlight text and have it read it out, then maybe, but I would say that that definitely falls under fair use. All right, so I guess that's it for the talk. I think we're hitting the timing mark really well. Thank you so much, David, for that. That was really, really interesting. I learned a lot. And thanks everyone for watching and stay on. I think there might be some news coming up. Thanks, and everyone.
 Alright, so again, let's introduce the next talk, Accessible Inputs for Readers, Coders  and Hackers, the talk by David Williams King about custom, well not off the shelf, but  custom accessibility solutions. He will give you some demonstrations and that includes his  own custom made voice input and eyelid blink system. Here is David Williams King. Thank you  for the introduction. Let's go ahead and get started. So yeah, I'm talking about accessibility,  particularly accessible input for readers, coders and hackers. So what do I mean by  accessibility? I mean people that have physical or motor impairments, this could be due to  repetitive strain injury, carpal tunnel, all kinds of medical conditions. If you have this type  of thing, you probably can't use a normal computer keyboard, computer mouse or even a phone touch  screen. However, technology does allow users to interact with these devices just using different  forms of input. And it's really valuable to these people because being able to interact with a device  provides some agency, they can do things on their own and it provides a means of communication with  the outside world. So it's an important problem to look at. And it's one I care about a lot. Let's  talk a bit about me for a moment. I'm a system security person. I did a PhD in cybersecurity at  Columbia. If you're interested in low level software defenses, you can look that up. And I'm  currently the CTO at a startup called Alpha Secure. I started developing medical issues in  around 2014. And as a result of that, in an ongoing fashion, I can only type a few thousand keystrokes  per day. Roughly 15,000 is my maximum. That sounds like a lot, but imagine you're typing at 100  words per minute. That's 500 characters per minute, which means it takes you 30 minutes to hit 15,000  characters. So essentially, I can work like the equivalent of a fast programmer for half an hour.  And then after that, I would be unable to use my hands for anything, including like, you know,  preparing food for myself or opening and closing doors and so on. So I have to be very careful  about my hand use and actually have a little program that you can see on the slide there that  measures the keystrokes for me so I can tell when I'm going over. So what do I do? Well, I do a lot  of pair programming for sure. I log into the same machine as other people and we work together. I'm  also a very heavy user of speech recognition. And I give a talk at that about voice coding with  speech recognition at the Hope 11 conference. So you can go check that out if you're interested. So  when I talk about accessible input, I mean different ways that a human can provide input to a  computer. So ergonomic keyboards are a simple one. Speech recognition, eye tracking or gaze tracking.  So you can see where you're looking or where you're pointing your head and maybe use that to  replace a mouse. That's head gestures, I suppose. And there's always this distinction between bespoke,  like custom input mechanisms and somewhat mainstream ones. So I'll give you some examples. You've  probably heard of Stephen Hawking. He's a very famous professor and he was actually a bit of an  extreme case. He was diagnosed with ALS when he was 21. So his physical abilities degraded over  the years because he lived for many decades after that. And he went through many communication  mechanisms. Initially, his speech changed so that it was only intelligible to his family and close  friends, but he was still able to speak. And then after that, he would work with a human interpreter  and raise his eyebrows to pick various letters. And then, keep in mind, this is like the 60s or  70s, right? So computers were not really where they are today. Later, he would operate a switch  with one hand, just like on off, on off, kind of Morse code and select from a bank of words. And  that was around 15 words per minute. Eventually, he was unable to move his hand. So a team of  engineers from Intel worked with him and they figured out they were trying to do like brain  scans and all kinds of stuff. But again, this was like in the 80s. So there was not, not too much  they could do. So they basically just created some custom software to detect muscle movements in his  cheek. And he used that with predictive, predictive words the same way that a phone,  smartphone keyboard will predict like which word you want to say next. Stephen Hawking used something  similar to that except instead of swiping on a phone, he was moving his, his cheek muscles. So  that's obviously a sequence of like highly customized input mechanisms for, for someone and  very, very specialized for that person. I also want to talk about someone else named Professor  Song Muk Lee, whom I've met that was me when I had more of a beard than I do now. He, he's a  professor at Seoul National University in South Korea. And he, he's sometimes called like the  Korean Stephen Hawking because he's a big advocate for people with disabilities and whatnot. Anyway,  what he uses is you can see a little orange device near his mouth there. He, it's called a sip and  puff mouse. So he can blow into it and suck air through it and also move it around. And that acts  as a mouse cursor on the Android device in front of him. It will move the cursor around and click  when he, when he blows air and so on. So that combined with speech recognition, lets him use  mainstream Android hardware. He still has access to, you know, email apps and like web browsers  and like maps and everything that comes on a normal Android device. So he's way more capable  than Stephen Hawking was because Stephen Hawking could communicate, but just to a person at a very  slow rate, right? Part of it is due to the nature of his injury, but it's also a testament to how  far the technology has improved. So let's talk a little bit about what makes good accessibility.  I think performance is very important, right? You want high accuracy, you don't want typos,  low latency, I don't want to speak and then five seconds later have words appear. It's too, it's  too long, especially if you have to make corrections, right? And you want high throughput, which we  already talked about. Oh, I forgot to mention Stephen Hawking had like, you know, 15 words per  minute. A normal person speaking is 150. So that's a big difference. The higher  throughput you can get the better. And for input accessibility, I think, and this is not  scientific, this is just what I've learned from using it myself and observing many of these systems.  I think it's important to get completeness, consistency and customization. For completeness,  I mean, can I do any action? So Stephen or Professor Song-Luk Lee, his orange mouth input device,  the Sip and Puff, is quite powerful, but it doesn't let him do every action. For example,  for some reason, when he gets an incoming call, the input doesn't work. So he has to call over a  person physically to like tap the accept call button or the reject call button, which is really  annoying, right? If you don't have completeness, you can't be fully independent. Consistency,  very important as well. The same way we develop motor memory for a muscle memory for a keyboard,  you develop memory for any types of patterns that you do. But if the thing you say or the thing  you do keeps changing in order to do the same action, that's not good. And finally, customization.  So the learning curve for beginners is important for any accessibility device, but designing for  expert use is almost more important because anyone who uses an accessibility interface becomes an  expert at it. The example I like to give is screen readers, like a blind person using a screen reader  on a phone, they will crank up the speed at which the speech is being produced. And I actually  met someone who made his speech 16 times faster than normal human speech. I could not understand it  at all. But he could understand it perfectly. And that's just because he used it so much that  he's become an expert at its use. Let's analyze ergonomic keyboards just for a moment because  it's fun. They are kind of like a normal keyboard. They'll have a slow pace when you're starting  to learn them. But once you're good at it, you have very good accuracy, like instantaneous low  latency, right? You press the key, the computer receives it immediately. And very high throughput,  as high as you are on a regular keyboard. So they're actually fantastic accessibility devices,  right? They're completely compatible with original keyboards. And if all you need is an  ergonomic keyboard, then you're in luck because it's a very good accessibility device. I'm going to  talk about two things, computers, but also Android devices. So let's start with Android devices.  Yes, the built-in voice recognition in Android is really incredible. So even though the microphones  on the devices aren't great, Google has just collected so much data from so many different  sources that they've built, like, better than human accuracy for their voice recognition. The voice  accessibility interface is kind of so-so. We'll talk about that in a bit. That's the interface  where you can control the Android device entirely by voice. For other input mechanisms, you could  use like a Sip and Puff device, or you could use physical styluses. That's something that I do a lot  actually, because for me, my fingers get sore. And if I can hold a stylus in my hand and kind of  not use my fingers, then that's, you know, very effective. So, and the Elacom styluses from a  Japanese company are the lightest I've found, and they don't require a lot of force. So the ones at  the top there are, they're like 12 grams, and the one at the bottom is 4.7 grams. And you've  required almost no force to use them, so very nice. On the left there, you can see the Android  speech recognition is built into the keyboard now, right? You can just press that and start  speaking in sports different languages, and it's very accurate. It's very nice. And actually,  when I was working at Google for a bit, I talked to the speech recognition team, and I was like,  why are you doing on-server speech recognition? You should do it on the devices. But of course,  Android devices are, they're all very different, and many of them are not very powerful, so they  were having trouble getting satisfactory speech recognition on the device. So for a long time,  there's some server latency, server lag, right? You do speech recognition and you wait a bit.  And then sometime this year, I just was using speech recognition, and it became  so much faster. I was extremely excited, and I looked into it, and yeah, they just switched  on my device at least, they switched on the on-device speech recognition model. And so now it's  incredibly fast and also incredibly accurate. I'm a huge fan of it. On the right-hand side,  we can actually see the voice access interface. So this is meant to allow you to use a phone  entirely by voice. Again, while I was at Google, I tried the beta version before it was publicly  released, and I was like, this is pretty bad. Mostly because it lacked completeness. There  would be things on the screen that would not be selected. So here we see show labels, and then I  can say like four, five, six, whatever, to tap on that thing. But as you can see at the bottom  there, there's like a Twitter web app link, and there's no number on it. So if I want to click on  that, I'm out of luck. And this is actually a problem in the design of the accessibility interface.  It doesn't expose the full DOM. It exposes only a subset of it, and so an accessibility mechanism  can't ever see those other things. And furthermore, the way the Google speech recognition works,  they have to reestablish a new connection every 30 seconds. And if you're in the middle of speaking,  it will just throw away whatever you were saying because it just decided it had to reconnect,  which is really unfortunate. They later released that publicly, and then sometime this year they  did an update, which is pretty nice. It now has like a mouse grid, which solves a lot of the  completeness problems. You can use a grid to narrow down somewhere on the screen and then tap  there. But the server issues and the expert use is still not good. Like, okay, if I want to do  something with a mouse grid, I have to say mouse grid on, six, five, mouse grid off. And I can't  combine those together. So there's a lot of latency. And it's not really that fun to use, but  better than nothing, absolutely. I just want to really briefly show you as well that this same  feature of like being able to select links on a screen is available on desktops. This is a plugin  for Chrome called Vimium. And it's very powerful because you can then combine this with keyboards  or other input mechanisms. And this one is complete. It uses the entire DOM and anything you can click  on will be highlighted. So very nice. I just want to give a quick example of me using some of these  systems. So I've been trying to learn Japanese. And there's a couple of highly regarded websites  for this, but they're not consistent when I use the browser show labels. Like, you know, the thing  to press next page or something like that, or like, you know, I give up or whatever is, it keeps  changing. So the letters that are being used keep changing. And that's because of the dynamic way  that they're generating the HTML. So not really very useful. What I do instead is I use a program  called Anki. And that has very simple shortcuts in his desktop app, one, two, three, four. So it's  nice to use and consistent. And it syncs with an Android app. And then I can use my stylus on the  Android device. So it works pretty well. But even so, you know, as you can see from the chart  in the bottom there, there are many days when I can't use this, even though I would like to,  because I've overused my hands or overused my voice. When I'm using voice recognition all day,  every day, I do tend to lose my voice. And as you can see from the graph, sometimes I lose it for  like a week or two at a time. So same thing with any accessibility interface, you know, you got to  use many different techniques. And it's always, it's never perfect. It's just the best you can do  at that moment. Something else I like to do is read books. I read a lot of books. And I love  eBook readers, you know, the dedicated eInk displays, you can read them in sunlight, they last  forever battery-wise. Unfortunately, it's hard to add like other input mechanisms to them. They  don't have microphones or other sensors. And you can't really install custom software on them.  But for Android-based devices, and they're also like eBook reading apps for Android devices,  they have everything. You can install custom software and they have microphones and many other  sensors. So I made two apps that allow you to read eBooks with an eBook reader. The first one is  Voice Next Page. It's based on one of my speech recognition engine called Silvius. And it does  do server-based recognition. So you have to capture all the audio, use 300 kilobits a second to send  it to the server and recognize things like Next Page, Previous Page. However, it doesn't cut out  every 30 seconds. It keeps going. So that's one win for it, I guess. And it is published in the  Play Store. Huge thanks to Sarah Leventhal who did a lot of the implementation, very complicated to  make an accessibility app on Android. But we persevered and works quite nicely. So I'm going  to actually show you an example of Voice Next Page. This is a... Over here, this is my phone on the  left-hand side. Just captured so that you guys can see it. So here's the Voice Next Page. And  basically, the connection is green. I can do the servers up and running and so on. I just press  Start and then I'll switch to an Android reading app and say Next Page, Previous Page. I won't  speak otherwise because it will chapel everything I'm saying. Next Page, Next Page, Previous Page.  Center, Center, Foreground. Stop listening. So that's a demo of the Voice Next Page. And it's  extremely helpful. I built it a couple of years ago along with Sarah. And I use it a lot. So yeah,  you can go ahead and download it if you guys want to try it out. And the other one is called Blink  Next Page. So the idea for this... I got this idea from a research paper this year that was  studying islet gestures. I didn't use any of their code, but it's a great idea. So the way this works  is you detect blinks by using the Android camera and then you can trigger an action like turning  pages in an eBook reader. This actually doesn't need any networking. It's able to use the on device  face recognition models from Google. And it is still under development, so it's not on the play  story yet. But it is working. And please contact me if you want to try it. So just give me one  moment to set that demo up here. And so I'm going to use... The main problem with this...  The main problem with this current implementation is that it uses two devices.  So that was easier to implement. And I use two devices anyway, but obviously I want a  one device version if I'm actually going to use it for anything. So here's how this works.  This device I point at me at my eyes. The other device I put wherever it's convenient to read.  Oops, sorry. And if I blink my eyes, the phone will buzz once it detects that I blink my eyes  and it will turn the page automatically on the other Android device. Now I have to blink both  my eyes for half a second. If I want to go backwards, I can blink just my left eye.  And if I want to go forwards, quickly, I can blink my right eye and hold it.  Anyway, it does have some false positives. That's why I like you can go backwards.  In case it detects that you've accidentally flipped the page. And lighting is also very important.  Like if I have a light behind me, then this is not going to be able to identify whether my eyes  are open or closed properly. So it has some limitations, but very simple to use. So I'm a big fan.  Okay, so that's enough about Android devices. Let's talk very briefly about desktop computers.  So if you're going to use a desktop computer, of course, try using that show labels plugin in a  browser for native apps, you can try Dragon Naturally Speaking, which is fine if you're just  using basic things. But if you're trying to do complicated things, you should definitely use  a voice coding system. You could also consider using eye tracking to replace a mouse. I personally,  I don't use that. I find it hurts my eyes, but I do use a track ball with very little force and  a walk on tablet. Some people will even scroll up and down by humming, for example, but I don't  have that set up. There's a bunch of nice talks out there on voice coding. The top left is Tavis  Reds talk from many years ago that got many of us interested. Emily Shia gave a talk there  about like best practices for voice coding. And then I gave a talk a couple years ago at the  Hope 11 conference, which you can also check out. It's mostly out of date by now, but it's still  interesting. So there are a lot of voice coding systems. The sort of grandfather of the mall is  Dragonfly. It's become a grammar standard. Caster is if you're willing to memorize lots of  unusual words, you can become much better, much faster than I currently am at voice coding.  A Nia is how you originally used Dragon to work on a Linux machine, for example, because Dragon only  runs on Windows. Talon is a closed source program, which is, but it's very, very powerful, has a  big user base, especially for Mac OS. There are ports now. And Talon used to use Dragon, but it's  now using a speech system from Facebook. Sylveus is the system that I created. The models are not  very accurate, but it's a nice architecture where there's client server, so it makes it easy to build  things like the voice next page. So the voice next page was using Sylveus. And then the most recent  one I think on this list is Calde Active Grammar, which is extremely powerful and extremely customizable,  and it's also open source. It works on all platforms, so I really highly recommend that. So let's talk  a bit more about Calde Active Grammar. But first, for voice coding, I've already mentioned you have  to be careful how you use your voice, right? Breathe through your belly. Don't tighten your  muscles and breathe from your chest. Try to speak normally. And I'm not particularly good at this.  Like, you'll hear me when I'm speaking commands. My inflection changes, so I do tend to overuse  my voice. But yeah, I just have to be conscious of that. The microphone hardware does matter. I do  recommend like a blue Yeti on a microphone arm that you can pull and put close to your face like  this. I'll use this one for my speaking demo. And yeah, and the other thing is your grammar is  fully customizable. So if you keep saying a word and the system doesn't recognize it, just change  it to another word. And it's complete in the sense you can type any key on the keyboard. And the most  important thing for expert use or customizability is that you can do chaining. So with a voice  coding system, you can say multiple commands at once. And it's a huge time savings. You'll see  what I mean when I give a quick demo. When I do voice coding, I'm a very heavy Vim and Tmux user.  You know, there have been, I've worked with many people before, so I have some cheat sheet information  there. So if you're interested, you can go check that out. But yeah, let's just do a quick demo  of voice coding here. Turn this mic on. Desk left to control delta, open new terminal,  Charlie, Delta space slash Tango mic papa enter command Vim, hotel, hotel point, Charlie, papa,  papa, enter. India hash word include space lango. India Oscar word stream wrangle, enter, enter.  Indian noitango space word main. Nope, my car to Indian noi. Landrun space lace enter,  enter race up tab. Word print Fox scratch nope. Code standard Charlie Oscar uniform Tango space  lango lango space quote sentence Hello voice coding bang. Scratch six Delta India, noi golf bang back  slash noi quote. semi colon act, Sky Fox mic Romeo noi Oscar word return space number zero semi colon  act. Vim save and quit. Golf plus plus space hotel hotel tab minus Oscar space hotel hotel.  Enter point slash hotel hotel enter. Desk right to.  So that's just a quick example of voice coding. You can use it to write any programming language.  You can use it to control anything on your desktop. It's very powerful. It has a bit of a learning  curve, but it's very powerful. So the creator of Calde active grammar is also named David.  I'm named David, but just a coincidence. And he says of Calde active grammar that  I haven't typed with a keyboard in many years and Calde active grammar is bootstrapped in that I  have been developing it entirely using the previous versions of it. So David has a medical condition  that means he has very low dexterity. So it's hard for him to use a keyboard. And yeah, he basically  got Calde active grammar working through the skin of his teeth or something and then continues to  develop it using it. And yeah, I'm a huge fan of the project. I haven't contributed much, but I did  give some of the harbor resources like GPU and CPU compute compute resources to allow training to  happen. But I would also like to show you a video of David using Calde active grammar just  so you can see it as well. So the other thing about David is that he has  he has a speech impediment or a speech, I don't know, an accent or whatever. So it's difficult  for a normal speech recognition system to understand him. And you might have trouble  understanding him here, but you can see in the lower right what the speech system understands  that he's saying. Oh, I realized that I do need to switch something in OBS so that you guys can  hear it. Sorry. There we go. Tim. Number one. Similar comments space. Number one, one, one.  Content line. Content, script, enter. Text. Timer assignment. Aposite. Content, enter two times.  And then, past rent space. And then,  Anyway, you get the idea. And hopefully you guys are able to hear that. If not, you can also find  this on the website that I'm going to show you at the end. Oh, one other thing I want to show you  about this is David has actually set up this humming to scroll, which I think is pretty cool.  Of course, I got and turned off the OBS there, but he's just going,  and it's understanding that and scrolling down. So something that I'm able to do with my  track ball, but that he's using his voice for. So pretty cool. So I'm almost done here in summary.  Good input accessibility means you need completeness, consistency, and customization. You need to be  able to do any action that you could do with the other input mechanisms. And doing the same input  should have the same action. And remember, your users will become experts. So the system needs  to be designed for that. For ebook reading. Yes, we're trying. I'm trying to allow anyone to read,  even if they're experiencing some severe physical or motor impairment, because I think  that gives you a lot of power to be able to turn the pages and read your favorite books.  And for speech recognition, yeah, Android speech recognition is very good.  Sylvia's accuracy is not so good, but it's easy to use quickly for experimentation and to make  other types of things like voice next page. And please do check out Caldea Active Grammar if you  have some serious need for voice recognition. Lastly, I put all of this onto a website,  voxhub.io. So you can see voice next page, link next page, Caldea Active Grammar,  and so on, just instructions for how to use it and how to set it up. So please do check that out.  And tons of acknowledgments, lots of people that have helped me along the way, but I want to especially  call out Professor Sang-Muk Lee, who actually invited me to Korea a couple of times to give talks,  a big inspiration. And of course, David Zuro has actually been able to bootstrap  into a fully voice coding environment. So that's all I have for today. Thank you very much.  Right. I suppose I'm back on the air. So let me see. I want to remind everyone before we go  into the Q&A that you can ask your questions for this talk on IRC. The link is under the video,  or you can use Twitter, or the FETIverse with the hashtag RC32. Again, I'll hold it up here, RC  number three, TWO. And wow, thanks for talking, David. That was really interesting. Thanks for  talking, David. I think we have a couple of questions from the Signal Angels. Before that,  I just want to say I recently spent some time playing with the voiceover system in iOS, and  that can now actually tell you what is on a photo, which is kind of amazing. Oh, by the way, I can't  hear you here on the mobile. Yeah, sorry. I wasn't saying anything. Yeah, no, it's so I focus mostly  on input accessibility, right, which is like, how do you get data to the computer? But there's been  huge improvements in the other way around as well, right, the computer? Yeah, so we have about, let's  see, five, six minutes left, at least for Q&A. We have a question by Toby plus plus, he asks,  your next page application looks cool. Do you have statistics of how many people use it or found it  on the app store? Not very many. The voice next page was advertised only so far as a little academic  poster. So I've gotten a few people to use it. But I run eight concurrent workers. And we've never  hit more than that. So not super, not super popular, but I do hope that some people will see it because  of this talk and go and check it out. Cool. Next question, how error prone are the speech  recognition systems at all? E.g. Can you do coding while doing workouts?  So one thing about speech recognition is very sensitive to the microphone. So when you're doing it, you  can't do it.  Any mistakes, right? That's the thing about having low latency, you just say something,  and you watch it, and you make sure that it was what you wanted to say. I don't know exactly how  many words per second, words per minute, I can say with voice coding, but I can say it much faster  than regular speech. So I would say at least like 200, maybe 300 words per minute. So it's  actually a very high bandwidth. Question from Pepe, JN, Divos. Any advice for software authors to  make their stuff more accessible? There are good web accessibility guidelines. So if you're just  making a website or something, I would definitely follow those. They tend to be focused more on  people that are blind because that is, you know, it's more of an obvious fail. Like they just can't  interact at all with your website. But things like, you know, if Duolingo, for example, had used  the same accessibility access tag on their like next button, then they would always be the same  letter for me. And I wouldn't have to be like, Fox, Charlie, Fox, Delta, Fox, something changes  all the time. So I think consistency is very important. And integrating with any existing  accessibility API is also very important, web APIs, Android APIs, and so on. Because, you know,  we can't make every program out there like voice compatible, we just have to meet in the middle  where they interact at the keyboard layer or the all American has a question. I wonder if these  systems use similar approaches like stenography with mnemonics, or if there's any projects working  having that in mind? A very good question. So the first thing everyone uses is the NATO phonetic  alphabet to spell letters, for example, alpha Bravo, Charlie. Some people then will substitute  letters for things that are too long, like November, I use no way. Sometimes the speech  system doesn't understand you. Whenever I said alpha, Dragon was like, Oh, you're saying offer.  So I had changed it. It's arch for me arch bravchar. So and also most of these grammars are in a  common grammar format. They are written in Python, and they're compatible with dragonfly. So you can  grab a grammar for, I don't know, for a Nia and get it to work with Caldeactive grammar with very  little effort. I actually have a grammar that works on both a Nia and Caldeactive grammar. And  that's what I use. So there's a bit of lingua franca, I guess you can kind of guess what other  people are using. But at the same time, there's a lot of customization, you know,  because people change words, they add their own commands, they change words based on what the  speech system understands. LEB asks, is there an online community you can propose for accessibility  technologies? There's a there's an amazing forum for anything related to voice coding.  All the developers of new voice coding software are there. Sorry, I just need to drink.  So it's a really fantastic resource. I do link to it from voxhub.io. I believe is at the bottom  of the Caldeactive grammar page. So you can definitely check that out. For general accessibility,  I don't know, I could recommend the accessibility mailing list at Google, but that's only if you  work at Google. Other than that. Yeah, I think it depends on your community, right? I think  if you're looking for web accessibility, you could go for some Mozilla mailing lists and so on. If  you're looking for desktop accessibility, then maybe you could go find some stuff about the  Windows speech API and the Windows accessibility. One last question from Joe Nielsen. Could there  be legal issues if you make an ebook into audio? I'm not sure what that refers to.  Yeah, so if you are like doing, if you're using a screen reader and you like, you know, try to get  it to read out the contents of a of an ebook, right? So most of the time there are fair use  exceptions for copyright law, even in the US. And making a copy of yourself for personal purposes  so that you can access it is usually considered fair use. If you were trying to commercialize it  or make money off of that or like, I don't know, you're a famous streamer and all you do is highlight  text and have it read it out, then maybe, but I would say that that definitely falls under fair use.  All right, so I guess that's it for the talk. I think we're hitting the timing mark really well.  Thank you so much, David, for that. That was really, really interesting. I learned a lot.  And thanks everyone for watching and stay on. I think there might be some news coming up. Thanks,  and everyone.
rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_0_47.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_48_143.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_144_191.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_192_239.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_240_287.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_288_1463.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_1464_4187.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_4188_5675.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_5676_11891.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_11892_13415.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_13416_24179.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_24180_24623.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_24624_24671.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_24672_24719.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_24720_24779.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_24780_25079.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_25080_25139.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_25140_25187.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_25188_25343.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_25344_25571.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_25572_26891.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_26892_27035.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_27036_27131.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_27132_27251.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_27252_27299.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_27300_28835.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_28836_34079.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_34080_34247.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_34248_34307.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_34308_35999.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_36000_37355.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_37356_37811.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_37812_37907.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_37908_38423.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_38424_38507.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_38508_39167.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_39168_39323.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_39324_39395.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_39396_39779.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_39780_42191.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_42192_45323.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_45324_45575.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_45576_45623.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_45624_45671.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_45672_45719.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_45720_45767.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_45768_45815.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_45816_45959.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_45960_53483.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_53484_54215.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_54216_54263.png rc3-mcr-11436-eng-deu-Accessible_input_for_readers_coders_and_hackers_hd_54264_54482.png
Alright, so again, let's introduce the next talk, Accessible Inputs for Readers, Coders and Hackers, the talk by David Williams King about custom, well not off the shelf, but custom accessibility solutions. He will give you some demonstrations and that includes his own custom made voice input and eyelid blink system. Here is David Williams King. Thank you for the introduction. Let's go ahead and get started. So yeah, I'm talking about accessibility, particularly accessible input for readers, coders and hackers. So what do I mean by accessibility? I mean people that have physical or motor impairments, this could be due to repetitive strain injury, carpal tunnel, all kinds of medical conditions. If you have this type of thing, you probably can't use a normal computer keyboard, computer mouse or even a phone touch screen. However, technology does allow users to interact with these devices just using different forms of input. And it's really valuable to these people because being able to interact with a device provides some agency, they can do things on their own and it provides a means of communication with the outside world. So it's an important problem to look at. And it's one I care about a lot. Let's talk a bit about me for a moment. I'm a system security person. I did a PhD in cybersecurity at Columbia. If you're interested in low level software defenses, you can look that up. And I'm currently the CTO at a startup called Alpha Secure. I started developing medical issues in around 2014. And as a result of that, in an ongoing fashion, I can only type a few thousand keystrokes per day. Roughly 15,000 is my maximum. That sounds like a lot, but imagine you're typing at 100 words per minute. That's 500 characters per minute, which means it takes you 30 minutes to hit 15,000 characters. So essentially, I can work like the equivalent of a fast programmer for half an hour. And then after that, I would be unable to use my hands for anything, including like, you know, preparing food for myself or opening and closing doors and so on. So I have to be very careful about my hand use and actually have a little program that you can see on the slide there that measures the keystrokes for me so I can tell when I'm going over. So what do I do? Well, I do a lot of pair programming for sure. I log into the same machine as other people and we work together. I'm also a very heavy user of speech recognition. And I give a talk at that about voice coding with speech recognition at the Hope 11 conference. So you can go check that out if you're interested. So when I talk about accessible input, I mean different ways that a human can provide input to a computer. So ergonomic keyboards are a simple one. Speech recognition, eye tracking or gaze tracking. So you can see where you're looking or where you're pointing your head and maybe use that to replace a mouse. That's head gestures, I suppose. And there's always this distinction between bespoke, like custom input mechanisms and somewhat mainstream ones. So I'll give you some examples. You've probably heard of Stephen Hawking. He's a very famous professor and he was actually a bit of an extreme case. He was diagnosed with ALS when he was 21. So his physical abilities degraded over the years because he lived for many decades after that. And he went through many communication mechanisms. Initially, his speech changed so that it was only intelligible to his family and close friends, but he was still able to speak. And then after that, he would work with a human interpreter and raise his eyebrows to pick various letters. And then, keep in mind, this is like the 60s or 70s, right? So computers were not really where they are today. Later, he would operate a switch with one hand, just like on off, on off, kind of Morse code and select from a bank of words. And that was around 15 words per minute. Eventually, he was unable to move his hand. So a team of engineers from Intel worked with him and they figured out they were trying to do like brain scans and all kinds of stuff. But again, this was like in the 80s. So there was not, not too much they could do. So they basically just created some custom software to detect muscle movements in his cheek. And he used that with predictive, predictive words the same way that a phone, smartphone keyboard will predict like which word you want to say next. Stephen Hawking used something similar to that except instead of swiping on a phone, he was moving his, his cheek muscles. So that's obviously a sequence of like highly customized input mechanisms for, for someone and very, very specialized for that person. I also want to talk about someone else named Professor Song Muk Lee, whom I've met that was me when I had more of a beard than I do now. He, he's a professor at Seoul National University in South Korea. And he, he's sometimes called like the Korean Stephen Hawking because he's a big advocate for people with disabilities and whatnot. Anyway, what he uses is you can see a little orange device near his mouth there. He, it's called a sip and puff mouse. So he can blow into it and suck air through it and also move it around. And that acts as a mouse cursor on the Android device in front of him. It will move the cursor around and click when he, when he blows air and so on. So that combined with speech recognition, lets him use mainstream Android hardware. He still has access to, you know, email apps and like web browsers and like maps and everything that comes on a normal Android device. So he's way more capable than Stephen Hawking was because Stephen Hawking could communicate, but just to a person at a very slow rate, right? Part of it is due to the nature of his injury, but it's also a testament to how far the technology has improved. So let's talk a little bit about what makes good accessibility. I think performance is very important, right? You want high accuracy, you don't want typos, low latency, I don't want to speak and then five seconds later have words appear. It's too, it's too long, especially if you have to make corrections, right? And you want high throughput, which we already talked about. Oh, I forgot to mention Stephen Hawking had like, you know, 15 words per minute. A normal person speaking is 150. So that's a big difference. The higher throughput you can get the better. And for input accessibility, I think, and this is not scientific, this is just what I've learned from using it myself and observing many of these systems. I think it's important to get completeness, consistency and customization. For completeness, I mean, can I do any action? So Stephen or Professor Song-Luk Lee, his orange mouth input device, the Sip and Puff, is quite powerful, but it doesn't let him do every action. For example, for some reason, when he gets an incoming call, the input doesn't work. So he has to call over a person physically to like tap the accept call button or the reject call button, which is really annoying, right? If you don't have completeness, you can't be fully independent. Consistency, very important as well. The same way we develop motor memory for a muscle memory for a keyboard, you develop memory for any types of patterns that you do. But if the thing you say or the thing you do keeps changing in order to do the same action, that's not good. And finally, customization. So the learning curve for beginners is important for any accessibility device, but designing for expert use is almost more important because anyone who uses an accessibility interface becomes an expert at it. The example I like to give is screen readers, like a blind person using a screen reader on a phone, they will crank up the speed at which the speech is being produced. And I actually met someone who made his speech 16 times faster than normal human speech. I could not understand it at all. But he could understand it perfectly. And that's just because he used it so much that he's become an expert at its use. Let's analyze ergonomic keyboards just for a moment because it's fun. They are kind of like a normal keyboard. They'll have a slow pace when you're starting to learn them. But once you're good at it, you have very good accuracy, like instantaneous low latency, right? You press the key, the computer receives it immediately. And very high throughput, as high as you are on a regular keyboard. So they're actually fantastic accessibility devices, right? They're completely compatible with original keyboards. And if all you need is an ergonomic keyboard, then you're in luck because it's a very good accessibility device. I'm going to talk about two things, computers, but also Android devices. So let's start with Android devices. Yes, the built-in voice recognition in Android is really incredible. So even though the microphones on the devices aren't great, Google has just collected so much data from so many different sources that they've built, like, better than human accuracy for their voice recognition. The voice accessibility interface is kind of so-so. We'll talk about that in a bit. That's the interface where you can control the Android device entirely by voice. For other input mechanisms, you could use like a Sip and Puff device, or you could use physical styluses. That's something that I do a lot actually, because for me, my fingers get sore. And if I can hold a stylus in my hand and kind of not use my fingers, then that's, you know, very effective. So, and the Elacom styluses from a Japanese company are the lightest I've found, and they don't require a lot of force. So the ones at the top there are, they're like 12 grams, and the one at the bottom is 4.7 grams. And you've required almost no force to use them, so very nice. On the left there, you can see the Android speech recognition is built into the keyboard now, right? You can just press that and start speaking in sports different languages, and it's very accurate. It's very nice. And actually, when I was working at Google for a bit, I talked to the speech recognition team, and I was like, why are you doing on-server speech recognition? You should do it on the devices. But of course, Android devices are, they're all very different, and many of them are not very powerful, so they were having trouble getting satisfactory speech recognition on the device. So for a long time, there's some server latency, server lag, right? You do speech recognition and you wait a bit. And then sometime this year, I just was using speech recognition, and it became so much faster. I was extremely excited, and I looked into it, and yeah, they just switched on my device at least, they switched on the on-device speech recognition model. And so now it's incredibly fast and also incredibly accurate. I'm a huge fan of it. On the right-hand side, we can actually see the voice access interface. So this is meant to allow you to use a phone entirely by voice. Again, while I was at Google, I tried the beta version before it was publicly released, and I was like, this is pretty bad. Mostly because it lacked completeness. There would be things on the screen that would not be selected. So here we see show labels, and then I can say like four, five, six, whatever, to tap on that thing. But as you can see at the bottom there, there's like a Twitter web app link, and there's no number on it. So if I want to click on that, I'm out of luck. And this is actually a problem in the design of the accessibility interface. It doesn't expose the full DOM. It exposes only a subset of it, and so an accessibility mechanism can't ever see those other things. And furthermore, the way the Google speech recognition works, they have to reestablish a new connection every 30 seconds. And if you're in the middle of speaking, it will just throw away whatever you were saying because it just decided it had to reconnect, which is really unfortunate. They later released that publicly, and then sometime this year they did an update, which is pretty nice. It now has like a mouse grid, which solves a lot of the completeness problems. You can use a grid to narrow down somewhere on the screen and then tap there. But the server issues and the expert use is still not good. Like, okay, if I want to do something with a mouse grid, I have to say mouse grid on, six, five, mouse grid off. And I can't combine those together. So there's a lot of latency. And it's not really that fun to use, but better than nothing, absolutely. I just want to really briefly show you as well that this same feature of like being able to select links on a screen is available on desktops. This is a plugin for Chrome called Vimium. And it's very powerful because you can then combine this with keyboards or other input mechanisms. And this one is complete. It uses the entire DOM and anything you can click on will be highlighted. So very nice. I just want to give a quick example of me using some of these systems. So I've been trying to learn Japanese. And there's a couple of highly regarded websites for this, but they're not consistent when I use the browser show labels. Like, you know, the thing to press next page or something like that, or like, you know, I give up or whatever is, it keeps changing. So the letters that are being used keep changing. And that's because of the dynamic way that they're generating the HTML. So not really very useful. What I do instead is I use a program called Anki. And that has very simple shortcuts in his desktop app, one, two, three, four. So it's nice to use and consistent. And it syncs with an Android app. And then I can use my stylus on the Android device. So it works pretty well. But even so, you know, as you can see from the chart in the bottom there, there are many days when I can't use this, even though I would like to, because I've overused my hands or overused my voice. When I'm using voice recognition all day, every day, I do tend to lose my voice. And as you can see from the graph, sometimes I lose it for like a week or two at a time. So same thing with any accessibility interface, you know, you got to use many different techniques. And it's always, it's never perfect. It's just the best you can do at that moment. Something else I like to do is read books. I read a lot of books. And I love eBook readers, you know, the dedicated eInk displays, you can read them in sunlight, they last forever battery-wise. Unfortunately, it's hard to add like other input mechanisms to them. They don't have microphones or other sensors. And you can't really install custom software on them. But for Android-based devices, and they're also like eBook reading apps for Android devices, they have everything. You can install custom software and they have microphones and many other sensors. So I made two apps that allow you to read eBooks with an eBook reader. The first one is Voice Next Page. It's based on one of my speech recognition engine called Silvius. And it does do server-based recognition. So you have to capture all the audio, use 300 kilobits a second to send it to the server and recognize things like Next Page, Previous Page. However, it doesn't cut out every 30 seconds. It keeps going. So that's one win for it, I guess. And it is published in the Play Store. Huge thanks to Sarah Leventhal who did a lot of the implementation, very complicated to make an accessibility app on Android. But we persevered and works quite nicely. So I'm going to actually show you an example of Voice Next Page. This is a... Over here, this is my phone on the left-hand side. Just captured so that you guys can see it. So here's the Voice Next Page. And basically, the connection is green. I can do the servers up and running and so on. I just press Start and then I'll switch to an Android reading app and say Next Page, Previous Page. I won't speak otherwise because it will chapel everything I'm saying. Next Page, Next Page, Previous Page. Center, Center, Foreground. Stop listening. So that's a demo of the Voice Next Page. And it's extremely helpful. I built it a couple of years ago along with Sarah. And I use it a lot. So yeah, you can go ahead and download it if you guys want to try it out. And the other one is called Blink Next Page. So the idea for this... I got this idea from a research paper this year that was studying islet gestures. I didn't use any of their code, but it's a great idea. So the way this works is you detect blinks by using the Android camera and then you can trigger an action like turning pages in an eBook reader. This actually doesn't need any networking. It's able to use the on device face recognition models from Google. And it is still under development, so it's not on the play story yet. But it is working. And please contact me if you want to try it. So just give me one moment to set that demo up here. And so I'm going to use... The main problem with this... The main problem with this current implementation is that it uses two devices. So that was easier to implement. And I use two devices anyway, but obviously I want a one device version if I'm actually going to use it for anything. So here's how this works. This device I point at me at my eyes. The other device I put wherever it's convenient to read. Oops, sorry. And if I blink my eyes, the phone will buzz once it detects that I blink my eyes and it will turn the page automatically on the other Android device. Now I have to blink both my eyes for half a second. If I want to go backwards, I can blink just my left eye. And if I want to go forwards, quickly, I can blink my right eye and hold it. Anyway, it does have some false positives. That's why I like you can go backwards. In case it detects that you've accidentally flipped the page. And lighting is also very important. Like if I have a light behind me, then this is not going to be able to identify whether my eyes are open or closed properly. So it has some limitations, but very simple to use. So I'm a big fan. Okay, so that's enough about Android devices. Let's talk very briefly about desktop computers. So if you're going to use a desktop computer, of course, try using that show labels plugin in a browser for native apps, you can try Dragon Naturally Speaking, which is fine if you're just using basic things. But if you're trying to do complicated things, you should definitely use a voice coding system. You could also consider using eye tracking to replace a mouse. I personally, I don't use that. I find it hurts my eyes, but I do use a track ball with very little force and a walk on tablet. Some people will even scroll up and down by humming, for example, but I don't have that set up. There's a bunch of nice talks out there on voice coding. The top left is Tavis Reds talk from many years ago that got many of us interested. Emily Shia gave a talk there about like best practices for voice coding. And then I gave a talk a couple years ago at the Hope 11 conference, which you can also check out. It's mostly out of date by now, but it's still interesting. So there are a lot of voice coding systems. The sort of grandfather of the mall is Dragonfly. It's become a grammar standard. Caster is if you're willing to memorize lots of unusual words, you can become much better, much faster than I currently am at voice coding. A Nia is how you originally used Dragon to work on a Linux machine, for example, because Dragon only runs on Windows. Talon is a closed source program, which is, but it's very, very powerful, has a big user base, especially for Mac OS. There are ports now. And Talon used to use Dragon, but it's now using a speech system from Facebook. Sylveus is the system that I created. The models are not very accurate, but it's a nice architecture where there's client server, so it makes it easy to build things like the voice next page. So the voice next page was using Sylveus. And then the most recent one I think on this list is Calde Active Grammar, which is extremely powerful and extremely customizable, and it's also open source. It works on all platforms, so I really highly recommend that. So let's talk a bit more about Calde Active Grammar. But first, for voice coding, I've already mentioned you have to be careful how you use your voice, right? Breathe through your belly. Don't tighten your muscles and breathe from your chest. Try to speak normally. And I'm not particularly good at this. Like, you'll hear me when I'm speaking commands. My inflection changes, so I do tend to overuse my voice. But yeah, I just have to be conscious of that. The microphone hardware does matter. I do recommend like a blue Yeti on a microphone arm that you can pull and put close to your face like this. I'll use this one for my speaking demo. And yeah, and the other thing is your grammar is fully customizable. So if you keep saying a word and the system doesn't recognize it, just change it to another word. And it's complete in the sense you can type any key on the keyboard. And the most important thing for expert use or customizability is that you can do chaining. So with a voice coding system, you can say multiple commands at once. And it's a huge time savings. You'll see what I mean when I give a quick demo. When I do voice coding, I'm a very heavy Vim and Tmux user. You know, there have been, I've worked with many people before, so I have some cheat sheet information there. So if you're interested, you can go check that out. But yeah, let's just do a quick demo of voice coding here. Turn this mic on. Desk left to control delta, open new terminal, Charlie, Delta space slash Tango mic papa enter command Vim, hotel, hotel point, Charlie, papa, papa, enter. India hash word include space lango. India Oscar word stream wrangle, enter, enter. Indian noitango space word main. Nope, my car to Indian noi. Landrun space lace enter, enter race up tab. Word print Fox scratch nope. Code standard Charlie Oscar uniform Tango space lango lango space quote sentence Hello voice coding bang. Scratch six Delta India, noi golf bang back slash noi quote. semi colon act, Sky Fox mic Romeo noi Oscar word return space number zero semi colon act. Vim save and quit. Golf plus plus space hotel hotel tab minus Oscar space hotel hotel. Enter point slash hotel hotel enter. Desk right to. So that's just a quick example of voice coding. You can use it to write any programming language. You can use it to control anything on your desktop. It's very powerful. It has a bit of a learning curve, but it's very powerful. So the creator of Calde active grammar is also named David. I'm named David, but just a coincidence. And he says of Calde active grammar that I haven't typed with a keyboard in many years and Calde active grammar is bootstrapped in that I have been developing it entirely using the previous versions of it. So David has a medical condition that means he has very low dexterity. So it's hard for him to use a keyboard. And yeah, he basically got Calde active grammar working through the skin of his teeth or something and then continues to develop it using it. And yeah, I'm a huge fan of the project. I haven't contributed much, but I did give some of the harbor resources like GPU and CPU compute compute resources to allow training to happen. But I would also like to show you a video of David using Calde active grammar just so you can see it as well. So the other thing about David is that he has he has a speech impediment or a speech, I don't know, an accent or whatever. So it's difficult for a normal speech recognition system to understand him. And you might have trouble understanding him here, but you can see in the lower right what the speech system understands that he's saying. Oh, I realized that I do need to switch something in OBS so that you guys can hear it. Sorry. There we go. Tim. Number one. Similar comments space. Number one, one, one. Content line. Content, script, enter. Text. Timer assignment. Aposite. Content, enter two times. And then, past rent space. And then, Anyway, you get the idea. And hopefully you guys are able to hear that. If not, you can also find this on the website that I'm going to show you at the end. Oh, one other thing I want to show you about this is David has actually set up this humming to scroll, which I think is pretty cool. Of course, I got and turned off the OBS there, but he's just going, and it's understanding that and scrolling down. So something that I'm able to do with my track ball, but that he's using his voice for. So pretty cool. So I'm almost done here in summary. Good input accessibility means you need completeness, consistency, and customization. You need to be able to do any action that you could do with the other input mechanisms. And doing the same input should have the same action. And remember, your users will become experts. So the system needs to be designed for that. For ebook reading. Yes, we're trying. I'm trying to allow anyone to read, even if they're experiencing some severe physical or motor impairment, because I think that gives you a lot of power to be able to turn the pages and read your favorite books. And for speech recognition, yeah, Android speech recognition is very good. Sylvia's accuracy is not so good, but it's easy to use quickly for experimentation and to make other types of things like voice next page. And please do check out Caldea Active Grammar if you have some serious need for voice recognition. Lastly, I put all of this onto a website, voxhub.io. So you can see voice next page, link next page, Caldea Active Grammar, and so on, just instructions for how to use it and how to set it up. So please do check that out. And tons of acknowledgments, lots of people that have helped me along the way, but I want to especially call out Professor Sang-Muk Lee, who actually invited me to Korea a couple of times to give talks, a big inspiration. And of course, David Zuro has actually been able to bootstrap into a fully voice coding environment. So that's all I have for today. Thank you very much. Right. I suppose I'm back on the air. So let me see. I want to remind everyone before we go into the Q&A that you can ask your questions for this talk on IRC. The link is under the video, or you can use Twitter, or the FETIverse with the hashtag RC32. Again, I'll hold it up here, RC number three, TWO. And wow, thanks for talking, David. That was really interesting. Thanks for talking, David. I think we have a couple of questions from the Signal Angels. Before that, I just want to say I recently spent some time playing with the voiceover system in iOS, and that can now actually tell you what is on a photo, which is kind of amazing. Oh, by the way, I can't hear you here on the mobile. Yeah, sorry. I wasn't saying anything. Yeah, no, it's so I focus mostly on input accessibility, right, which is like, how do you get data to the computer? But there's been huge improvements in the other way around as well, right, the computer? Yeah, so we have about, let's see, five, six minutes left, at least for Q&A. We have a question by Toby plus plus, he asks, your next page application looks cool. Do you have statistics of how many people use it or found it on the app store? Not very many. The voice next page was advertised only so far as a little academic poster. So I've gotten a few people to use it. But I run eight concurrent workers. And we've never hit more than that. So not super, not super popular, but I do hope that some people will see it because of this talk and go and check it out. Cool. Next question, how error prone are the speech recognition systems at all? E.g. Can you do coding while doing workouts? So one thing about speech recognition is very sensitive to the microphone. So when you're doing it, you can't do it. Any mistakes, right? That's the thing about having low latency, you just say something, and you watch it, and you make sure that it was what you wanted to say. I don't know exactly how many words per second, words per minute, I can say with voice coding, but I can say it much faster than regular speech. So I would say at least like 200, maybe 300 words per minute. So it's actually a very high bandwidth. Question from Pepe, JN, Divos. Any advice for software authors to make their stuff more accessible? There are good web accessibility guidelines. So if you're just making a website or something, I would definitely follow those. They tend to be focused more on people that are blind because that is, you know, it's more of an obvious fail. Like they just can't interact at all with your website. But things like, you know, if Duolingo, for example, had used the same accessibility access tag on their like next button, then they would always be the same letter for me. And I wouldn't have to be like, Fox, Charlie, Fox, Delta, Fox, something changes all the time. So I think consistency is very important. And integrating with any existing accessibility API is also very important, web APIs, Android APIs, and so on. Because, you know, we can't make every program out there like voice compatible, we just have to meet in the middle where they interact at the keyboard layer or the all American has a question. I wonder if these systems use similar approaches like stenography with mnemonics, or if there's any projects working having that in mind? A very good question. So the first thing everyone uses is the NATO phonetic alphabet to spell letters, for example, alpha Bravo, Charlie. Some people then will substitute letters for things that are too long, like November, I use no way. Sometimes the speech system doesn't understand you. Whenever I said alpha, Dragon was like, Oh, you're saying offer. So I had changed it. It's arch for me arch bravchar. So and also most of these grammars are in a common grammar format. They are written in Python, and they're compatible with dragonfly. So you can grab a grammar for, I don't know, for a Nia and get it to work with Caldeactive grammar with very little effort. I actually have a grammar that works on both a Nia and Caldeactive grammar. And that's what I use. So there's a bit of lingua franca, I guess you can kind of guess what other people are using. But at the same time, there's a lot of customization, you know, because people change words, they add their own commands, they change words based on what the speech system understands. LEB asks, is there an online community you can propose for accessibility technologies? There's a there's an amazing forum for anything related to voice coding. All the developers of new voice coding software are there. Sorry, I just need to drink. So it's a really fantastic resource. I do link to it from voxhub.io. I believe is at the bottom of the Caldeactive grammar page. So you can definitely check that out. For general accessibility, I don't know, I could recommend the accessibility mailing list at Google, but that's only if you work at Google. Other than that. Yeah, I think it depends on your community, right? I think if you're looking for web accessibility, you could go for some Mozilla mailing lists and so on. If you're looking for desktop accessibility, then maybe you could go find some stuff about the Windows speech API and the Windows accessibility. One last question from Joe Nielsen. Could there be legal issues if you make an ebook into audio? I'm not sure what that refers to. Yeah, so if you are like doing, if you're using a screen reader and you like, you know, try to get it to read out the contents of a of an ebook, right? So most of the time there are fair use exceptions for copyright law, even in the US. And making a copy of yourself for personal purposes so that you can access it is usually considered fair use. If you were trying to commercialize it or make money off of that or like, I don't know, you're a famous streamer and all you do is highlight text and have it read it out, then maybe, but I would say that that definitely falls under fair use. All right, so I guess that's it for the talk. I think we're hitting the timing mark really well. Thank you so much, David, for that. That was really, really interesting. I learned a lot. And thanks everyone for watching and stay on. I think there might be some news coming up. Thanks, and everyone.
accessibility solutions. He will give you some made voice input and eyelid blink system. Here is particularly accessible input for readers, coders strain injury, carpal tunnel, all kinds of keyboard, computer mouse or even a phone touch interact with these devices just using different people because being able to interact with a that, I would be unable to use my hands for very heavy user of speech recognition. And I give one. Speech recognition, eye tracking or gaze that to replace a mouse. That's head gestures, I he went through many communication mechanisms. code and select from a bank of words. And that was around 15 words per minute. Eventually, he engineers from Intel worked with him and they some custom software to detect muscle movements in his cheek. And he used that with predictive, recognition, lets him use mainstream Android hardware. He still has access to, you know, email because Stephen Hawking could communicate, but Stephen Hawking had like, you know, 15 words per memory for a keyboard, you develop memory for any order to do the same action, that's not good. And The example I like to give is screen readers, like a blind person using a screen reader on a computers, but also Android devices. So let's to the speech recognition team, and I was like, speech recognition works, they have to know, as you can see from the chart in the bottom using voice recognition all day, every day, I do everything. You can install custom software and with an eBook reader. The first one is Voice Next to make an accessibility app on Android. But we listening. So that's a demo of the Voice Next recognition models from Google. And it is still backwards. In case it detects that you've going to use a desktop computer, of course, try voice coding system. You could also consider extremely customizable, and it's also open fully customizable. So if you keep saying a word write any programming language. You can use it to powerful. It has a bit of a learning curve, but David has a medical condition that means he has it's difficult for a normal speech recognition system to understand him. And you might have customization. You need to be able to do any designed for that. For ebook reading. Yes, we're workers. And we've never hit more than that. So exactly how many words per second, words per because people change words, they add their own