Mainlining the reMarkable 2 eInk tablet
Francis, Alistair
The reMarkable 2 is an eInk tablet, based on the i.MX7 SoC. The tablet ships with a fork of the 4.14 kernel and a custom rootFS built with OpenEmbedded. The vendor kernel is based on the NXP vendor kernel with a large collection of rM2 specific patches on top. This talk discusses the process of adding support for the rM2 to the mainline Linux kernel. The talk first discusses the process of understanding the original boot process and then getting access to a download mechanism and serial console. Finally the talk covers the process to start upstreaming the patches from a vendor kernel to the mainline kernel.
 Hello, everyone. My name is Alstair and I'm here today to talk about mainlining the Remarkable 2 E Ink tablet. So before I start, I'd just like to point out that this is a personal project that I have done in my spare time and no way related to Remarkable or any other company. This is just a personal hobby project. And today I'm going to talk about my experiences in upstreaming and the support for the E Ink tablet to the Linux kernel. I have some technical details but nothing too intense. So if anyone has any questions or you'd like more information, just ask in the Q&A or afterwards feel free to shoot me an email or message me on matrix. So the Remarkable 2, which I have here, is a 10-inch E Ink tablet. So E Ink is commonly found in things like cogos and kindles. It's a pretty cool technology that allows you to display an image or the screen without any power. So basically the way it works is you kind of push an image to it. It changes these stuff inside it, not going on in detail too much. And then it will stay there without any interference. So they're commonly used in books because for every example, you only use power when you actually turn the page or change the display. And like a normal computer screen, it's not refreshing, you know, numbers of times a second. So they're good for low power things like that. And they're pretty cool tablets. So it has a touchscreen. The device has a touchscreen, a Waycon stylus, Wi-Fi and an IMX7D, so dual core SOC. So for a consumer product, Remarkable is pretty open actually. So it does have proprietary user space software and that's kind of where they market. So it has all the features they talk about. So this is where you import a PDF and you can draw on it and highlight it, things like that. You can doodle and draw sketches and all those type of things as well. They advertise it as a kind of replacement for paper. So you can draw on it and treat it like a book. You can write notes and things like that. It's not an e-book. It doesn't have a library store like a Kindle or a Kobo would, but it's a pretty cool device. So like I said, they're pretty open software. So it ships to the 4.14 kernel, which was later updated to the 5.4 kernel. It's all built with open and embedded. But you have root SSH access to the device. So there's no lockdowns on that. And there's also no lockdown on the boot up, right? So there's no secure boot that doesn't mean you can't change your images and there's no debug and restrictions. So you have full access to the device. You might call even a bind by GPI license requirements and release the kernel and you boot source. Not every company does that. So it's nice to see them doing the right thing. This is all available on GitHub and you have the full Git log, which is also really nice. They even have a developer community built around them. So not from the company itself, but there are communities that design different launches, different applications. It has co-reader support, which is an ebook kind of software, open source software. And there's even a community maintaining a package management system built in OPKG, which is a very small package system. So it's all pretty cool. In saying that, though, they're remarkable, not very atrium friendly. They have an open thing they've ever submitted, any patches upstream. They're seeing uninterested in doing that. They also don't take patches to their own trees from what I've seen. So before I purchase one, I dug into the software status. I'm the type of person who uses a pine phone. I don't want to buy something and then get stuck with it, you know, running out of vendor support, having known vulnerabilities or the vendor changing on you and enforcing some sort of subscription service. So I dug into it and looked at what they were doing. So like I previously mentioned, I could see the kernel source and I figured out there's about 300 patches on top of the NXP vendor kernel from Remarkable. I also found that people have ported full Linux to previous, the previous one, the Remarkable, just the one I guess, but people have ported a full Linux distro. I had that up and running. They still use the vendor kernel, but there was interest in this type of thing. Now, the first one even had UI access over USB, so you can kind of see what was going on. So at that point, I decided, you know, I'd get one and investigate if I could upstream these 300 patches. And that was in December of 2020, so a bit over a year ago. So initial investigation. After I got the device, I wanted to know how it worked and what it was doing. So with my root SSH access, I started digging. So I figured out, like I mentioned, it's a 4.14 kernel and it's all built with open embedded. I could even figure out what meta layers they used. Unfortunately, we didn't have access or the open source community doesn't have access to all of those layers. And I did ask them, but they never got back to me. They use an AB update mechanism in U-POOT. So there are two root partitions, two root of S, two boot and one home partition. And so the way this works, you can find the scripts actually in the root of S. It's an update, they flash like a new boot and a new root of S, and then reboot, set some flags and reboot and reboot into that. This is really exciting because it meant that if I messed up the kernel boot, it wouldn't be stuck with a brick, but hopefully it would fall back to the previous partition. I also checked some FCC photos, which is what this picture on the slide is from, to see if there are any obvious U-Out ports. I couldn't find anything, but I didn't want to open up my own device anyway. I thought it was still worth looking in case it was obvious and maybe it was worth it. So now that I knew how the device worked, I wanted to start my own kernel work. So first I needed serial access. In order to do that, in order to do any kernel work, I figured I need to be able to see what it's doing. So when it works, that's great. I can SSH into it and see the DM message and whatever else I need. But what if it doesn't boot, what if it doesn't come up? I didn't need to know why. I need some sort of something. So I don't have debug access, that seems a little ambitious to ask for, but at least I wanted some U-Out access so I could see what's happening. Unfortunately, there are remarkable two who doesn't have serial access over USB. So that was a real downer. This is a large blocker. So without U-Out access, I wouldn't be able to see what's going on. So I didn't want to open the device, but luckily after digging through the kernel, sorry, just grabbing it, digging through the kernel code, it's out of my case. I figured out what seems to be that serial 5 and USB are both mixed to the Pogo pins. So it's these. That's all I have to see. But there were four, five pins on the side. Yeah, that was kind of useless. Five pins on the side. And it's the type of thing you'd attach an external keyboard accessory to. And so after double checking that my AB update was there and it's working, I made the plunge and edited the device tree on the host and SSH into the device and I synced it across. But I edited the device tree to change the U-Out to U-Out 5, rebooted and saw what happened. So luckily the device booted and I could see U-Out. So you see here in this picture, it's a little hard to see, but these are the five U-Out Pogo pins, sorry, five Pogo pins. And I've got connectors on them and I've got debug probes connected to two of them to get ground and TX out. So there's no U-Out in, so I can't send commands in, but I can receive messages from the remarkable and I can get the entire boot log from the kernel at this point. So next step was to flash the kernel. I couldn't spend the entire time building the kernel on my host, r-syncing it across, rebooting, seeing what happened, and then if that did work, that's great, but if it didn't work, hoping that the AB update still worked this time. That seemed too slow, too tedious. It was going to probably damage the flash with all these writes and wasn't really a workable solution. So luckily, there are other people in the community have figured out there is a recovery access, recovery mode. So using a resistor on two of the USB-C pins, you could short them and that would tell you, boot, to jump into the wrong download mode. And so by doing that, you could then use UU3U, which is a standard IMX tool to download recovery images. So this meant in this picture, so I have my Pogo pins like I had before, which plug into my computer and I can use my USB and then from my computer to download images with the same two debug probes connected to them to get serial out as well, while also using USB-C, extension cable, to break out two of the USB-C to then short a resistor across two of them. So with this setup, some sticky tape and a little bit of luck in holding the connectors, I could download images too remarkable, UGroot, Kernel, RooTFS, and Vystre, reboot and see the entire log. So now that I can boot a kernel and build myself, I want to do two things. First, I want to get a more recent kind of known working state. So I took the latest 5.4 NXP kernel, which is the latest kernel NXP I've released at this point. It was in a 5.10. And started porting the remarkable two features onto it. So this was about the 300 patches that I set originally. So I didn't port all, I kind of squashed them together into chunks. So they weren't ready to upstream, but they were kind of subdivided. So for example, I had like drivers input was one, add drivers input, add drivers input, top screen, things like that. So they're big chunks, but I think I got about down to about 30 patches at this point. And this worked. I could actually boot the remarkable, not just boot, I had it kind of up and running. This is not in the picture, I'll come to the picture a little bit. But at this point, it was actually working enough that after a few changes, I swapped across to using this full time. So I downloaded this onto the device, replaced the boot images with my own. And this is what I was doing for day to day use. So this is great. I had a nice working state. Instead of comparing against the 5.14, sorry, the 4.14 kernel, I was comparing against the 5.4 kernel now for my future work, which is much more recent at the time, I think, Linux would have been 5.12 or something, 5.13. So not that recent, but still better than 4.14. This is actually the first 5.4 kernel to run remarkable. The company itself updated later. But we are so yet the community was first to run the 5.4 kernel, at least publicly. And the 5.4 kernel used still by the human is still newer than the one released by the company. So that's all interesting. So the next step was mainline. So now that I had the NXP kernel working, I wanted to get mainline working. So I thought, how hard can it really be, right? Someone must have upstreamed all these NXP patches at this point. So it should just work. I mean, obviously not the remarkable part, but I thought that everything else would be okay. So I sat down, looked at some of the other IonX boards and kind of based a device tree up there with the minimum I could get away with, just basically you are nothing else. I also at this point thought, you know, I don't want to do, I don't want to boot this up with my, and use my home, my RoutaVest on my home directory on this thing. And something goes wrong, corrupts it and, and, you know, I can't get stuck back. So I thought also what I need is I need a RAM vest. I need something running in RAM that I can use instead. And so that's what open and better came in. So I knew open and better works. I use open and better all the time. It's a great, great distro tool then. And so I thought, you know, add a machine to that and use that. So there's a patch there. It's actually upstreamed as well. So the meta free scale third party machine comp. And that builds remarkable to kernel, U-boot, device tree and RoutaVest. And so I was using that. And with that, I did manage to boot just mainline. So mainline was just a device tree on top. I had UR and nothing else, no screen, no input, no nothing, but not, not, not even any Wi-Fi, but it was working. And that started the process of, of mainlining. So those patches were sent upstream, have since been merged. And that was kind of the first step. So finally in this, this bring up process, I thought, I need Wi-Fi. So like I mentioned earlier, there's no TA, there's no RX to the device. So I can't send a command to it. So I thought, well, you know, if, if once I get to use this space, what if I want to, you know, test something, I want to check something, I'm going to need Wi-Fi SSH at this point. So that's not too bad because the Wi-Fi driver was already upstream. So it's in the command line kernel. Okay, easy, right? I thought that would be the easy one. So the actual patch, five lines and device free patches to enable Wi-Fi and good to go. Unfortunately, this is probably one of the longest processes in the whole thing. So at first it wouldn't work because I didn't have the right firmware to the firmware version at the remarkable binary ships with an older one than what the kernel expects. So I tried to upstream that firmware to, and this is the same firmware actually the Raspberry Pi and the Raspberry Pi is used. So I tried to upstream that to the kernel firmware because I could get into my open embedded root of s. But that didn't go too well. Some license testing issues there. So eventually I just gave up and thought, you know what, I'm just going to boot and use the root of s on the device. How bad can it really be? Luckily, it wasn't that bad. But this then meant I compiled everything without modules. So I just put them the Wi-Fi driver into the kernel. That didn't work because that meant it tried to load it before the root of s was available and couldn't find the firmware. So eventually I figured that out. That's for people who do kernel development a lot. That's kind of obvious. But for people who don't, I don't have that much experience with Linux development. It took me a little bit to figure it out. So once I had that running, that was okay. So I could just copy modules across. I thought, you know, I'll sync them across. We're in the working setup and then I can reboot. And it worked a little bit, but it was really hidden next. I would add a debug statement and would just not work. And then I would remove it and it wouldn't work. And then add it and it would work and it couldn't figure it out what was going on. Eventually I realized the problem is that I was running the 5.4 custom kernel at this point and it doesn't have shut down support. I don't know why, but that's one of the things I never got working. So I can't turn it off. So I just hold the power button and force it off. And when I did that, it wouldn't always sync or flush everything to the EMMC. And so depending on the state, the kernel modules wouldn't work. And so eventually I figured out I have to make sure I run sync and flush everything to the internal EMMC before I power it off and reboot. And so eventually when I figured that out, combined with the different firmware versions and building the kernel modules, and actually I needed some device, some K-config changes, I had Wi-Fi working reliably and set my 5 line patch so I think I had run 2 months ago at this point or something. So that was a little embarrassing. So the Wacom. So I've got it here actually. So the Wacom style works kind of in real time. It's a bit hard to see, but let's see how this goes. So you can see it kind of draws as I draw it. So that's the latest, the 5.16.0. That's the patches I ended up with. So you can see here on the right-hand side of the screen, the patches from law. So this is the version 10 of my 12 patch series to add Wacom support. So this is a bit of a slog to get this going, but eventually at this point there were engineers from Wacom who were interested in seeing it merged as well. And this was all to add with the driver's touch screen, sorry, driver's input touch screen. There's a Wacom underscore s-grade c in there and that was this one. That was adding, improving the support there. And this is what the vendor did, so the remarkable guy did. So I thought that's the best place to be. Version 10 was finally nacked after all this time. So what happened in version 10 is the maintainers realized that the new Wacom's actually have full HID support. HID's a generic input interface, kind of pioneering input-screen Microsoft that really simplifies input. And the new Wacom ones support this. And as soon as they realized that, rightly so, they were rightly so. But they nacked my patches and said no, you have to do that. So at this point, it's deleted everything that I worked on and came back to it and ended up with basically two patches, one mostly just device tree, to add Wacom support to the remarkable. So it was already in there. I could just utilize all of the HID input framework. So in future, it's probably worth thinking about if there's generic frameworks you can use before you go run now and write your special custom driver. So the only problem with that was that the remarkable input is actually kind of rotated around 180 degrees. So the X and Y inputs had to be inverted. And in the Wacom driver, that's easy, you know, you have control over everything. But in the generic HID input, it's a little more difficult. So it took me a while to understand how the events worked and where this should actually go and what value we should be negating. But after that, it's still also difficult to modify the stuff because it affects lots of other things. So anyone who does have experience with drivers, you know, drivers input touchscreen knows there's always touch invert X, Y, device tree properties. So throw that in the dump. But like I said, though, here wasn't it wasn't implemented. So I ended up implementing that. So now we have support for that in in HID input and the ispread C HID. And that's what we use in the remarkable. So we can now flip them and negate the X and Y values, which works perfectly. As you just saw, you can now draw around and exactly matches where I expected to match on the remarkable. Here's a picture I took of FOSDEM logo on the remarkable at home. So like I kind of mentioned earlier, eating some pretty cool technology, you know, only updates when it only uses power when you when you update. And but one of the things about it is actually quite complex to use. And so because of this, you see most products use an IMX because it has a hard way in built called EPDC, which is that eating display controller, basically. And this is what kindles and cobers and all sorts of things use. And the remarkable uses it to the remarkable, though, is different in that they don't actually use this hardware. They instead just connect the screen directly to the LCD controller. And what that means is there's no way for the hardware to manage the waveforms or any of the other complex stuff related to eating. So I'm not going to too much detail about what that is. There's someone else that follows them dev room is going to talk about about driving eating displays. I'll leave that up to them. But basically, for the remarkable, it means there's a complex proprietary user space app that does manage all the waveforms. And it writes that data just to a frame buffer device. And the kernel, it has an MxS, FB, which is just a frame buffer by device for the IMAX. And that's what it uses. So this works great in the vendor panel. And looking at it, there even was that this device had been upstreamed and was upstreamed in the mainline kernel. Great. So they had us have to pull over some vendor patches and be good to go. But then I realized what actually happened is the vent, this frame buffer device, this MxS, FB, has been removed from mainline. So they had it and they removed it because it's been replaced with a DRM, a DRM driver. And that's what everyone's using upstream now. Unfortunately, the proprietary user space software doesn't work with the DRM driver. And I have no way to change it. So we're stuck using the frame buffer device. So no worries, we can just emulate the frame buffer device, right? So I tried that, have to write this down on my notes. But as a FBI, I put the screen info Ioptal that the proprietary user space uses. And that's trying to change some of the screen resolutions. And that's not supported in emulated frame buffer device. So we can't actually use this. So what we have is we have this new great DRM driver that should work. It doesn't because the proprietary user space doesn't support it. So in the end, I have to just backport, basically revert this MxS FB driver. And that's what I'm using today. And that's what I'm using my demo just then. And that's what I'm currently using. But that's not really main lineable. So there's two options. So obviously, we could extend frame buffer emulation to improve that to then support this new Ioptal and changing the screen resolutions. The other option is just remove the proprietary user space software. And there's an effort on going, not by me, but by others in the community to reverse engineer the e-ainting driving system and run an open source one. So this would be great because then we'd have a fully open source stack. And we could also then add DRM support to this software. So we don't even need frame buffer. We can just use a DRM directly. And so I'm hoping that's what's going to happen. And I have more details in the links at the end of the presentation too. So as you kind of go along, the way I tested things has changed. So initially, I would have that set up from the start with the upside down on the chair, and I would flash a kernel, make sure it booted. And then, for example, testing the wake home, I'd pick it up and draw a picture and make sure it worked. And then when I was done, I'd turn it off and reboot back to my custom 5.4 kernel that I kind of started with and go about using that. So eventually, at this point now, with about the 5.16, 5.17 kernel and extra patches on top, not just mainline, but at a point where we're going to daily drive it. So I'm now using this thing daily with the mainline, the mainline-inch kernel. And that's when you start to notice lots of other bugs that you probably wouldn't notice just in this quick testing turnaround. And so one of them is that I would put the device to sleep, walk away, and come back later. And it would be out of batteries if it was completely flat. But the weird thing with eating is that when the battery just dies or the machine just turns off, you see what it was lost doing because it doesn't clear the display. And so I could see that it was on the lock screen. And so first, it's really confusing because you try and touch it and it doesn't react and it eventually takes you a while to realize it's turned off. But once you get around that, I realized that it was, I thought it must be waking up, right, and putting it to sleep and something's happening, some interrupt somewhere, probably, you know, the touch screen is waking up and it's draining the batteries because I have sleep disabled. So once it wakes up, I'm going to go back to sleep unless you press the button. So okay, so that must be it. So eventually I was watching it one day and I realized what's actually happened, it's not what's happening, what's actually happening is after it goes to sleep for a few minutes, it would reboot, the whole thing would turn off and turn back on again. And then when it turns on, it gets to the lock screen and just sits there because like I said, I have sleep disabled, automatic sleep, and it will just burn through the battery and turn off. So okay, so it's reboot, that's a strange one. So I was like, okay, what could be maybe is a system D is crashing and it's causing a reboot that's possible, I guess, because I'm running their root of S and with my kernel. Or the other option is it's a watchdog timer and there's a watchdog rebooting. So I had a quick look at some of the logs and didn't really think it was the first one, system D one. So I thought, you know, it's probably a watchdog issue. And so I started digging into the kernel. And yeah, okay, there's a watchdog timer support, there's a driver for it and everything. Okay, when it goes to suspend, it sets the timer for two minutes. That's the maximum you can do about two minutes. That's the maximum you can do on 9 and X. And I looked at it and I was like, that's about when it turns up, it waits about two minutes in every boot. Okay, so it must be a watchdog timer. The watchdog timer is kicking in. But this is a mainline device, right? I didn't write this, this is already there. So does that mean everyone else has this device or every other IMX device reboots two minutes after going to sleep? That didn't seem right. So it's really through the specs, trying to figure out what's going on. And eventually I realized that although the watchdog timer cannot be disabled by software, so once you enable it, it can't be disabled. It does get disabled by hardware when you go into low power mode. So when you go into low power mode, the hardware will automatically kind of suspend the timer and you won't reboot. So why isn't that happening now? And so I dug into a little bit more and realized that there's no low power mode for the IMX7 or the IMX7D and not in mainline. So the vendor kernel works fine because they have low power mode implemented and they use that. So they don't have to worry about it. I guess other IMX devices are using low power mode too or don't suspend. And so they don't have to worry about it. But in this case, it don't have low power and I'm suspending. So eventually the watchdog times out and reboots the system. So I thought, okay, there must be a way for like a way to tell the kernel not to do that. I wanted to ping the watchdog every two minutes, looking at it for less than two minutes to keep it, make sure the watchdog time doesn't reboot on me. But there is a way. The kernel can do that. In fact, in the IMX driver, we're specifically telling the kernel not to do that. We're saying don't ping on suspend. And so that's because everyone else goes into low power mode and they don't need to. So for them, it can be a way to see a battery to wake up to ping this watchdog timer that's being disabled. So there's now patches sent and maybe by the time you read this, that we merge, but patches are on the list to add watchdog timer ping on suspend support for certain devices like the remarkable two. So now we can ping the watchdog on suspend. So we're not going to go to sleep. So that's a type of example of like a debugging issue or hit, not just from plugging in and testing it at your desk. We actually start carrying around and using it and putting it to sleep and leaving it for long periods of time, those two minutes isn't that long, but yet leaving it and using it and daily driving it. So 5.17 kernel. So the 5.16 kernel was released just for recent, not a little few weeks ago, I guess a week or two ago before I record this. So the 5.17 kernel is a little bit off, but the 5.17 kernel will have full boot support like I've talked about, Wi-Fi support and the way you can touch screen support. So that's pretty cool. The patches on list for the Syllogy NFD. So that's the power regulator to control the touch screen, control the screen, the LCD support, the LCD driver support, the cypress touch screen, which is like the finger touch screen. Now the really cool thing about that actually is that the same touch screen used in the Pine note, which is another eReader announced with Pine64. So some people have actually reached out to me about getting that to work on the Pine note. So the drive is in a process of being upstream and it needs a few changes for Pine note support, but it has been tested on the remarkable end of one of the co-votes and works on both of those. And the IMX Watchdog ping support that I just talked about is also has patches on list. So the next steps, so the current patches on list merged, get a toll tech package merged. So that's the remarkable package manager system. So I have actually Linux 5.16 support as a package in there. So I need someone to test that and merge that. So if anyone has a remarkable two-year-longster test, that's a great place to go. Get low power mode working. I've kind of talked about that a little bit. Fix up some paper cut bugs. There's a few kind of smaller issues. And reverse engineer the EEN driver like I talked about earlier. There's a link to the waived project. If anyone's interested, that'd be a great effort. And so that's it from me. I'm right at the end of my time. So I'm going to drop off now and I'll be around to insert any.
 Hello, everyone. My name is Alstair and I'm here today to talk about mainlining the Remarkable  2 E Ink tablet. So before I start, I'd just like to point out that this is a personal  project that I have done in my spare time and no way related to Remarkable or any other  company. This is just a personal hobby project. And today I'm going to talk about my experiences  in upstreaming and the support for the E Ink tablet to the Linux kernel. I have some technical  details but nothing too intense. So if anyone has any questions or you'd like more information,  just ask in the Q&A or afterwards feel free to shoot me an email or message me on matrix.  So the Remarkable 2, which I have here, is a 10-inch E Ink tablet. So E Ink is commonly  found in things like cogos and kindles. It's a pretty cool technology that allows you to  display an image or the screen without any power. So basically the way it works is you  kind of push an image to it. It changes these stuff inside it, not going on in detail too  much. And then it will stay there without any interference. So they're commonly used  in books because for every example, you only use power when you actually turn the page  or change the display. And like a normal computer screen, it's not refreshing, you know, numbers  of times a second. So they're good for low power things like that. And they're pretty  cool tablets. So it has a touchscreen. The device has a touchscreen, a Waycon stylus,  Wi-Fi and an IMX7D, so dual core SOC. So for a consumer product, Remarkable is pretty  open actually. So it does have proprietary user space software and that's kind of where  they market. So it has all the features they talk about. So this is where you import a  PDF and you can draw on it and highlight it, things like that. You can doodle and draw  sketches and all those type of things as well. They advertise it as a kind of replacement  for paper. So you can draw on it and treat it like a book. You can write notes and things  like that. It's not an e-book. It doesn't have a library store like a Kindle or a Kobo  would, but it's a pretty cool device. So like I said, they're pretty open software.  So it ships to the 4.14 kernel, which was later updated to the 5.4 kernel. It's all  built with open and embedded. But you have root SSH access to the device. So there's  no lockdowns on that. And there's also no lockdown on the boot up, right? So there's  no secure boot that doesn't mean you can't change your images and there's no debug  and restrictions. So you have full access to the device. You might call even a bind  by GPI license requirements and release the kernel and you boot source. Not every company  does that. So it's nice to see them doing the right thing. This is all available on  GitHub and you have the full Git log, which is also really nice. They even have a developer  community built around them. So not from the company itself, but there are communities  that design different launches, different applications. It has co-reader support, which  is an ebook kind of software, open source software. And there's even a community maintaining  a package management system built in OPKG, which is a very small package system. So it's  all pretty cool. In saying that, though, they're remarkable, not very atrium friendly. They  have an open thing they've ever submitted, any patches upstream. They're seeing uninterested  in doing that. They also don't take patches to their own trees from what I've seen.  So before I purchase one, I dug into the software status. I'm the type of person who uses a  pine phone. I don't want to buy something and then get stuck with it, you know, running  out of vendor support, having known vulnerabilities or the vendor changing on you and enforcing  some sort of subscription service. So I dug into it and looked at what they were doing.  So like I previously mentioned, I could see the kernel source and I figured out there's  about 300 patches on top of the NXP vendor kernel from Remarkable. I also found that  people have ported full Linux to previous, the previous one, the Remarkable, just the  one I guess, but people have ported a full Linux distro. I had that up and running. They  still use the vendor kernel, but there was interest in this type of thing. Now, the first  one even had UI access over USB, so you can kind of see what was going on. So at that  point, I decided, you know, I'd get one and investigate if I could upstream these 300  patches. And that was in December of 2020, so a bit over a year ago. So initial investigation.  After I got the device, I wanted to know how it worked and what it was doing. So with my  root SSH access, I started digging. So I figured out, like I mentioned, it's a 4.14 kernel  and it's all built with open embedded. I could even figure out what meta layers they  used. Unfortunately, we didn't have access or the open source community doesn't have  access to all of those layers. And I did ask them, but they never got back to me. They  use an AB update mechanism in U-POOT. So there are two root partitions, two root of S, two  boot and one home partition. And so the way this works, you can find the scripts actually  in the root of S. It's an update, they flash like a new boot and a new root of S, and then  reboot, set some flags and reboot and reboot into that. This is really exciting because  it meant that if I messed up the kernel boot, it wouldn't be stuck with a brick, but hopefully  it would fall back to the previous partition. I also checked some FCC photos, which is what  this picture on the slide is from, to see if there are any obvious U-Out ports. I couldn't  find anything, but I didn't want to open up my own device anyway. I thought it was still  worth looking in case it was obvious and maybe it was worth it. So now that I knew how the  device worked, I wanted to start my own kernel work. So first I needed serial access. In  order to do that, in order to do any kernel work, I figured I need to be able to see what  it's doing. So when it works, that's great. I can SSH into it and see the DM message and  whatever else I need. But what if it doesn't boot, what if it doesn't come up? I didn't  need to know why. I need some sort of something. So I don't have debug access, that seems a  little ambitious to ask for, but at least I wanted some U-Out access so I could see  what's happening. Unfortunately, there are remarkable two who doesn't have serial access  over USB. So that was a real downer. This is a large blocker. So without U-Out access,  I wouldn't be able to see what's going on. So I didn't want to open the device, but  luckily after digging through the kernel, sorry, just grabbing it, digging through the  kernel code, it's out of my case. I figured out what seems to be that serial 5 and USB  are both mixed to the Pogo pins. So it's these. That's all I have to see. But there  were four, five pins on the side. Yeah, that was kind of useless. Five pins on the side.  And it's the type of thing you'd attach an external keyboard accessory to. And so after  double checking that my AB update was there and it's working, I made the plunge and edited  the device tree on the host and SSH into the device and I synced it across. But I edited  the device tree to change the U-Out to U-Out 5, rebooted and saw what happened. So luckily  the device booted and I could see U-Out. So you see here in this picture, it's a little  hard to see, but these are the five U-Out Pogo pins, sorry, five Pogo pins. And I've  got connectors on them and I've got debug probes connected to two of them to get ground  and TX out. So there's no U-Out in, so I can't send commands in, but I can receive messages  from the remarkable and I can get the entire boot log from the kernel at this point. So  next step was to flash the kernel. I couldn't spend the entire time building the kernel  on my host, r-syncing it across, rebooting, seeing what happened, and then if that did  work, that's great, but if it didn't work, hoping that the AB update still worked this  time. That seemed too slow, too tedious. It was going to probably damage the flash with  all these writes and wasn't really a workable solution. So luckily, there are other people  in the community have figured out there is a recovery access, recovery mode. So using  a resistor on two of the USB-C pins, you could short them and that would tell you, boot,  to jump into the wrong download mode. And so by doing that, you could then use UU3U,  which is a standard IMX tool to download recovery images. So this meant in this picture, so  I have my Pogo pins like I had before, which plug into my computer and I can use my USB  and then from my computer to download images with the same two debug probes connected to  them to get serial out as well, while also using USB-C, extension cable, to break out  two of the USB-C to then short a resistor across two of them. So with this setup, some sticky  tape and a little bit of luck in holding the connectors, I could download images too remarkable,  UGroot, Kernel, RooTFS, and Vystre, reboot and see the entire log. So now that I can  boot a kernel and build myself, I want to do two things. First, I want to get a more  recent kind of known working state. So I took the latest 5.4 NXP kernel, which is the latest  kernel NXP I've released at this point. It was in a 5.10. And started porting the remarkable  two features onto it. So this was about the 300 patches that I set originally. So I didn't  port all, I kind of squashed them together into chunks. So they weren't ready to upstream,  but they were kind of subdivided. So for example, I had like drivers input was one, add drivers  input, add drivers input, top screen, things like that. So they're big chunks, but I think  I got about down to about 30 patches at this point. And this worked. I could actually boot  the remarkable, not just boot, I had it kind of up and running. This is not in the picture,  I'll come to the picture a little bit. But at this point, it was actually working enough  that after a few changes, I swapped across to using this full time. So I downloaded this  onto the device, replaced the boot images with my own. And this is what I was doing  for day to day use. So this is great. I had a nice working state. Instead of comparing  against the 5.14, sorry, the 4.14 kernel, I was comparing against the 5.4 kernel now  for my future work, which is much more recent at the time, I think, Linux would have been  5.12 or something, 5.13. So not that recent, but still better than 4.14. This is actually  the first 5.4 kernel to run remarkable. The company itself updated later. But we are so  yet the community was first to run the 5.4 kernel, at least publicly. And the 5.4 kernel  used still by the human is still newer than the one released by the company. So that's  all interesting. So the next step was mainline. So now that I had the NXP kernel working,  I wanted to get mainline working. So I thought, how hard can it really be, right? Someone  must have upstreamed all these NXP patches at this point. So it should just work. I mean,  obviously not the remarkable part, but I thought that everything else would be okay. So I sat  down, looked at some of the other IonX boards and kind of based a device tree up there with  the minimum I could get away with, just basically you are nothing else. I also at this point  thought, you know, I don't want to do, I don't want to boot this up with my, and use my home,  my RoutaVest on my home directory on this thing. And something goes wrong, corrupts  it and, and, you know, I can't get stuck back. So I thought also what I need is I need a  RAM vest. I need something running in RAM that I can use instead. And so that's what open  and better came in. So I knew open and better works. I use open and better all the time.  It's a great, great distro tool then. And so I thought, you know, add a machine to that  and use that. So there's a patch there. It's actually upstreamed as well. So the meta free  scale third party machine comp. And that builds remarkable to kernel, U-boot, device tree and  RoutaVest. And so I was using that. And with that, I did manage to boot just mainline.  So mainline was just a device tree on top. I had UR and nothing else, no screen, no input,  no nothing, but not, not, not even any Wi-Fi, but it was working. And that started the process  of, of mainlining. So those patches were sent upstream, have since been merged. And that  was kind of the first step. So finally in this, this bring up process, I thought, I need Wi-Fi.  So like I mentioned earlier, there's no TA, there's no RX to the device. So I can't send  a command to it. So I thought, well, you know, if, if once I get to use this space, what  if I want to, you know, test something, I want to check something, I'm going to need  Wi-Fi SSH at this point. So that's not too bad because the Wi-Fi driver was already upstream.  So it's in the command line kernel. Okay, easy, right? I thought that would be the easy  one. So the actual patch, five lines and device free patches to enable Wi-Fi and good to go.  Unfortunately, this is probably one of the longest processes in the whole thing. So at  first it wouldn't work because I didn't have the right firmware to the firmware version  at the remarkable binary ships with an older one than what the kernel expects. So I tried  to upstream that firmware to, and this is the same firmware actually the Raspberry Pi  and the Raspberry Pi is used. So I tried to upstream that to the kernel firmware because  I could get into my open embedded root of s. But that didn't go too well. Some license  testing issues there. So eventually I just gave up and thought, you know what, I'm just  going to boot and use the root of s on the device. How bad can it really be? Luckily,  it wasn't that bad. But this then meant I compiled everything without modules. So I just put  them the Wi-Fi driver into the kernel. That didn't work because that meant it tried to  load it before the root of s was available and couldn't find the firmware. So eventually  I figured that out. That's for people who do kernel development a lot. That's kind of  obvious. But for people who don't, I don't have that much experience with Linux development.  It took me a little bit to figure it out. So once I had that running, that was okay.  So I could just copy modules across. I thought, you know, I'll sync them across. We're in  the working setup and then I can reboot. And it worked a little bit, but it was really  hidden next. I would add a debug statement and would just not work. And then I would  remove it and it wouldn't work. And then add it and it would work and it couldn't figure  it out what was going on. Eventually I realized the problem is that I was running the 5.4  custom kernel at this point and it doesn't have shut down support. I don't know why,  but that's one of the things I never got working. So I can't turn it off. So I just hold the  power button and force it off. And when I did that, it wouldn't always sync or flush  everything to the EMMC. And so depending on the state, the kernel modules wouldn't work.  And so eventually I figured out I have to make sure I run sync and flush everything  to the internal EMMC before I power it off and reboot. And so eventually when I figured  that out, combined with the different firmware versions and building the kernel modules,  and actually I needed some device, some K-config changes, I had Wi-Fi working reliably and set  my 5 line patch so I think I had run 2 months ago at this point or something. So that was  a little embarrassing. So the Wacom. So I've got it here actually. So the Wacom style works  kind of in real time. It's a bit hard to see, but let's see how this goes. So you can see  it kind of draws as I draw it. So that's the latest, the 5.16.0. That's the patches I ended  up with. So you can see here on the right-hand side of the screen, the patches from law.  So this is the version 10 of my 12 patch series to add Wacom support. So this is a bit of  a slog to get this going, but eventually at this point there were engineers from Wacom  who were interested in seeing it merged as well. And this was all to add with the driver's  touch screen, sorry, driver's input touch screen. There's a Wacom underscore s-grade  c in there and that was this one. That was adding, improving the support there. And this  is what the vendor did, so the remarkable guy did. So I thought that's the best place  to be. Version 10 was finally nacked after all this time. So what happened in version  10 is the maintainers realized that the new Wacom's actually have full HID support. HID's  a generic input interface, kind of pioneering input-screen Microsoft that really simplifies  input. And the new Wacom ones support this. And as soon as they realized that, rightly  so, they were rightly so. But they nacked my patches and said no, you have to do that.  So at this point, it's deleted everything that I worked on and came back to it and ended  up with basically two patches, one mostly just device tree, to add Wacom support to  the remarkable. So it was already in there. I could just utilize all of the HID input  framework. So in future, it's probably worth thinking about if there's generic frameworks  you can use before you go run now and write your special custom driver. So the only problem  with that was that the remarkable input is actually kind of rotated around 180 degrees.  So the X and Y inputs had to be inverted. And in the Wacom driver, that's easy, you  know, you have control over everything. But in the generic HID input, it's a little more  difficult. So it took me a while to understand how the events worked and where this should  actually go and what value we should be negating. But after that, it's still also difficult  to modify the stuff because it affects lots of other things. So anyone who does have experience  with drivers, you know, drivers input touchscreen knows there's always touch invert X, Y, device  tree properties. So throw that in the dump. But like I said, though, here wasn't it wasn't  implemented. So I ended up implementing that. So now we have support for that in in HID  input and the ispread C HID. And that's what we use in the remarkable. So we can now flip  them and negate the X and Y values, which works perfectly. As you just saw, you can  now draw around and exactly matches where I expected to match on the remarkable. Here's  a picture I took of FOSDEM logo on the remarkable at home. So like I kind of mentioned earlier,  eating some pretty cool technology, you know, only updates when it only uses power when you  when you update. And but one of the things about it is actually quite complex to use.  And so because of this, you see most products use an IMX because it has a hard way in built  called EPDC, which is that eating display controller, basically. And this is what kindles  and cobers and all sorts of things use. And the remarkable uses it to the remarkable, though,  is different in that they don't actually use this hardware. They instead just connect  the screen directly to the LCD controller. And what that means is there's no way for  the hardware to manage the waveforms or any of the other complex stuff related to eating.  So I'm not going to too much detail about what that is. There's someone else that follows  them dev room is going to talk about about driving eating displays. I'll leave that up  to them. But basically, for the remarkable, it means there's a complex proprietary user  space app that does manage all the waveforms. And it writes that data just to a frame buffer  device. And the kernel, it has an MxS, FB, which is just a frame buffer by device for  the IMAX. And that's what it uses. So this works great in the vendor panel. And looking  at it, there even was that this device had been upstreamed and was upstreamed in the  mainline kernel. Great. So they had us have to pull over some vendor patches and be good  to go. But then I realized what actually happened is the vent, this frame buffer device, this  MxS, FB, has been removed from mainline. So they had it and they removed it because it's  been replaced with a DRM, a DRM driver. And that's what everyone's using upstream now.  Unfortunately, the proprietary user space software doesn't work with the DRM driver.  And I have no way to change it. So we're stuck using the frame buffer device. So no worries,  we can just emulate the frame buffer device, right? So I tried that, have to write this  down on my notes. But as a FBI, I put the screen info Ioptal that the proprietary user  space uses. And that's trying to change some of the screen resolutions. And that's not  supported in emulated frame buffer device. So we can't actually use this. So what we  have is we have this new great DRM driver that should work. It doesn't because the proprietary  user space doesn't support it. So in the end, I have to just backport, basically revert  this MxS FB driver. And that's what I'm using today. And that's what I'm using my demo just  then. And that's what I'm currently using. But that's not really main lineable. So there's  two options. So obviously, we could extend frame buffer emulation to improve that to  then support this new Ioptal and changing the screen resolutions. The other option is  just remove the proprietary user space software. And there's an effort on going, not by me,  but by others in the community to reverse engineer the e-ainting driving system and  run an open source one. So this would be great because then we'd have a fully open source  stack. And we could also then add DRM support to this software. So we don't even need frame  buffer. We can just use a DRM directly. And so I'm hoping that's what's going to happen.  And I have more details in the links at the end of the presentation too.  So as you kind of go along, the way I tested things has changed. So initially, I would  have that set up from the start with the upside down on the chair, and I would flash a kernel,  make sure it booted. And then, for example, testing the wake home, I'd pick it up and  draw a picture and make sure it worked. And then when I was done, I'd turn it off and  reboot back to my custom 5.4 kernel that I kind of started with and go about using that.  So eventually, at this point now, with about the 5.16, 5.17 kernel and extra patches on  top, not just mainline, but at a point where we're going to daily drive it. So I'm now  using this thing daily with the mainline, the mainline-inch kernel. And that's when  you start to notice lots of other bugs that you probably wouldn't notice just in this  quick testing turnaround. And so one of them is that I would put the device to sleep, walk  away, and come back later. And it would be out of batteries if it was completely flat.  But the weird thing with eating is that when the battery just dies or the machine just  turns off, you see what it was lost doing because it doesn't clear the display. And  so I could see that it was on the lock screen. And so first, it's really confusing because  you try and touch it and it doesn't react and it eventually takes you a while to realize  it's turned off. But once you get around that, I realized that it was, I thought it must  be waking up, right, and putting it to sleep and something's happening, some interrupt  somewhere, probably, you know, the touch screen is waking up and it's draining the batteries  because I have sleep disabled. So once it wakes up, I'm going to go back to sleep unless  you press the button. So okay, so that must be it. So eventually I was watching it one  day and I realized what's actually happened, it's not what's happening, what's actually  happening is after it goes to sleep for a few minutes, it would reboot, the whole thing  would turn off and turn back on again. And then when it turns on, it gets to the lock  screen and just sits there because like I said, I have sleep disabled, automatic sleep,  and it will just burn through the battery and turn off. So okay, so it's reboot, that's  a strange one. So I was like, okay, what could be maybe is a system D is crashing and it's  causing a reboot that's possible, I guess, because I'm running their root of S and with  my kernel. Or the other option is it's a watchdog timer and there's a watchdog rebooting.  So I had a quick look at some of the logs and didn't really think it was the first one,  system D one. So I thought, you know, it's probably a watchdog issue. And so I started  digging into the kernel. And yeah, okay, there's a watchdog timer support, there's a driver  for it and everything. Okay, when it goes to suspend, it sets the timer for two minutes.  That's the maximum you can do about two minutes. That's the maximum you can do on 9 and X.  And I looked at it and I was like, that's about when it turns up, it waits about two minutes  in every boot. Okay, so it must be a watchdog timer. The watchdog timer is kicking in. But  this is a mainline device, right? I didn't write this, this is already there. So does  that mean everyone else has this device or every other IMX device reboots two minutes  after going to sleep? That didn't seem right. So it's really through the specs, trying to  figure out what's going on. And eventually I realized that although the watchdog timer  cannot be disabled by software, so once you enable it, it can't be disabled. It does get  disabled by hardware when you go into low power mode. So when you go into low power  mode, the hardware will automatically kind of suspend the timer and you won't reboot.  So why isn't that happening now? And so I dug into a little bit more and realized that  there's no low power mode for the IMX7 or the IMX7D and not in mainline. So the vendor  kernel works fine because they have low power mode implemented and they use that. So they  don't have to worry about it. I guess other IMX devices are using low power mode too or  don't suspend. And so they don't have to worry about it. But in this case, it don't have  low power and I'm suspending. So eventually the watchdog times out and reboots the system.  So I thought, okay, there must be a way for like a way to tell the kernel not to do that.  I wanted to ping the watchdog every two minutes, looking at it for less than two minutes to  keep it, make sure the watchdog time doesn't reboot on me. But there is a way. The kernel  can do that. In fact, in the IMX driver, we're specifically telling the kernel not to do  that. We're saying don't ping on suspend. And so that's because everyone else goes into  low power mode and they don't need to. So for them, it can be a way to see a battery  to wake up to ping this watchdog timer that's being disabled. So there's now patches sent  and maybe by the time you read this, that we merge, but patches are on the list to add  watchdog timer ping on suspend support for certain devices like the remarkable two. So  now we can ping the watchdog on suspend. So we're not going to go to sleep. So that's  a type of example of like a debugging issue or hit, not just from plugging in and testing  it at your desk. We actually start carrying around and using it and putting it to sleep  and leaving it for long periods of time, those two minutes isn't that long, but yet leaving  it and using it and daily driving it. So 5.17 kernel. So the 5.16 kernel was released just  for recent, not a little few weeks ago, I guess a week or two ago before I record this.  So the 5.17 kernel is a little bit off, but the 5.17 kernel will have full boot support  like I've talked about, Wi-Fi support and the way you can touch screen support. So that's  pretty cool. The patches on list for the Syllogy NFD. So that's the power regulator to control  the touch screen, control the screen, the LCD support, the LCD driver support, the cypress  touch screen, which is like the finger touch screen. Now the really cool thing about that  actually is that the same touch screen used in the Pine note, which is another eReader  announced with Pine64. So some people have actually reached out to me about getting that  to work on the Pine note. So the drive is in a process of being upstream and it needs  a few changes for Pine note support, but it has been tested on the remarkable end of one  of the co-votes and works on both of those. And the IMX Watchdog ping support that I just  talked about is also has patches on list. So the next steps, so the current patches  on list merged, get a toll tech package merged. So that's the remarkable package manager system.  So I have actually Linux 5.16 support as a package in there. So I need someone to test  that and merge that. So if anyone has a remarkable two-year-longster test, that's a great place  to go. Get low power mode working. I've kind of talked about that a little bit. Fix up  some paper cut bugs. There's a few kind of smaller issues. And reverse engineer the EEN  driver like I talked about earlier. There's a link to the waived project. If anyone's  interested, that'd be a great effort. And so that's it from me. I'm right at the end  of my time. So I'm going to drop off now and I'll be around to insert any.
mobile_kernel_tablet_0_59.png mobile_kernel_tablet_60_107.png mobile_kernel_tablet_108_155.png mobile_kernel_tablet_156_251.png mobile_kernel_tablet_252_1163.png mobile_kernel_tablet_1164_5591.png mobile_kernel_tablet_5592_7187.png mobile_kernel_tablet_7188_9299.png mobile_kernel_tablet_9300_12371.png mobile_kernel_tablet_12372_14735.png mobile_kernel_tablet_14736_24395.png mobile_kernel_tablet_24396_29255.png mobile_kernel_tablet_29256_34703.png mobile_kernel_tablet_34704_42455.png mobile_kernel_tablet_42456_44063.png mobile_kernel_tablet_44064_44111.png mobile_kernel_tablet_44112_44159.png mobile_kernel_tablet_44160_44459.png mobile_kernel_tablet_44460_44651.png mobile_kernel_tablet_44652_44711.png mobile_kernel_tablet_44712_44831.png mobile_kernel_tablet_44832_44891.png mobile_kernel_tablet_44892_44963.png mobile_kernel_tablet_44964_45059.png mobile_kernel_tablet_45060_45107.png mobile_kernel_tablet_45108_45179.png mobile_kernel_tablet_45180_45347.png mobile_kernel_tablet_45348_45467.png mobile_kernel_tablet_45468_45469.png
Hello, everyone. My name is Alstair and I'm here today to talk about mainlining the Remarkable 2 E Ink tablet. So before I start, I'd just like to point out that this is a personal project that I have done in my spare time and no way related to Remarkable or any other company. This is just a personal hobby project. And today I'm going to talk about my experiences in upstreaming and the support for the E Ink tablet to the Linux kernel. I have some technical details but nothing too intense. So if anyone has any questions or you'd like more information, just ask in the Q&A or afterwards feel free to shoot me an email or message me on matrix. So the Remarkable 2, which I have here, is a 10-inch E Ink tablet. So E Ink is commonly found in things like cogos and kindles. It's a pretty cool technology that allows you to display an image or the screen without any power. So basically the way it works is you kind of push an image to it. It changes these stuff inside it, not going on in detail too much. And then it will stay there without any interference. So they're commonly used in books because for every example, you only use power when you actually turn the page or change the display. And like a normal computer screen, it's not refreshing, you know, numbers of times a second. So they're good for low power things like that. And they're pretty cool tablets. So it has a touchscreen. The device has a touchscreen, a Waycon stylus, Wi-Fi and an IMX7D, so dual core SOC. So for a consumer product, Remarkable is pretty open actually. So it does have proprietary user space software and that's kind of where they market. So it has all the features they talk about. So this is where you import a PDF and you can draw on it and highlight it, things like that. You can doodle and draw sketches and all those type of things as well. They advertise it as a kind of replacement for paper. So you can draw on it and treat it like a book. You can write notes and things like that. It's not an e-book. It doesn't have a library store like a Kindle or a Kobo would, but it's a pretty cool device. So like I said, they're pretty open software. So it ships to the 4.14 kernel, which was later updated to the 5.4 kernel. It's all built with open and embedded. But you have root SSH access to the device. So there's no lockdowns on that. And there's also no lockdown on the boot up, right? So there's no secure boot that doesn't mean you can't change your images and there's no debug and restrictions. So you have full access to the device. You might call even a bind by GPI license requirements and release the kernel and you boot source. Not every company does that. So it's nice to see them doing the right thing. This is all available on GitHub and you have the full Git log, which is also really nice. They even have a developer community built around them. So not from the company itself, but there are communities that design different launches, different applications. It has co-reader support, which is an ebook kind of software, open source software. And there's even a community maintaining a package management system built in OPKG, which is a very small package system. So it's all pretty cool. In saying that, though, they're remarkable, not very atrium friendly. They have an open thing they've ever submitted, any patches upstream. They're seeing uninterested in doing that. They also don't take patches to their own trees from what I've seen. So before I purchase one, I dug into the software status. I'm the type of person who uses a pine phone. I don't want to buy something and then get stuck with it, you know, running out of vendor support, having known vulnerabilities or the vendor changing on you and enforcing some sort of subscription service. So I dug into it and looked at what they were doing. So like I previously mentioned, I could see the kernel source and I figured out there's about 300 patches on top of the NXP vendor kernel from Remarkable. I also found that people have ported full Linux to previous, the previous one, the Remarkable, just the one I guess, but people have ported a full Linux distro. I had that up and running. They still use the vendor kernel, but there was interest in this type of thing. Now, the first one even had UI access over USB, so you can kind of see what was going on. So at that point, I decided, you know, I'd get one and investigate if I could upstream these 300 patches. And that was in December of 2020, so a bit over a year ago. So initial investigation. After I got the device, I wanted to know how it worked and what it was doing. So with my root SSH access, I started digging. So I figured out, like I mentioned, it's a 4.14 kernel and it's all built with open embedded. I could even figure out what meta layers they used. Unfortunately, we didn't have access or the open source community doesn't have access to all of those layers. And I did ask them, but they never got back to me. They use an AB update mechanism in U-POOT. So there are two root partitions, two root of S, two boot and one home partition. And so the way this works, you can find the scripts actually in the root of S. It's an update, they flash like a new boot and a new root of S, and then reboot, set some flags and reboot and reboot into that. This is really exciting because it meant that if I messed up the kernel boot, it wouldn't be stuck with a brick, but hopefully it would fall back to the previous partition. I also checked some FCC photos, which is what this picture on the slide is from, to see if there are any obvious U-Out ports. I couldn't find anything, but I didn't want to open up my own device anyway. I thought it was still worth looking in case it was obvious and maybe it was worth it. So now that I knew how the device worked, I wanted to start my own kernel work. So first I needed serial access. In order to do that, in order to do any kernel work, I figured I need to be able to see what it's doing. So when it works, that's great. I can SSH into it and see the DM message and whatever else I need. But what if it doesn't boot, what if it doesn't come up? I didn't need to know why. I need some sort of something. So I don't have debug access, that seems a little ambitious to ask for, but at least I wanted some U-Out access so I could see what's happening. Unfortunately, there are remarkable two who doesn't have serial access over USB. So that was a real downer. This is a large blocker. So without U-Out access, I wouldn't be able to see what's going on. So I didn't want to open the device, but luckily after digging through the kernel, sorry, just grabbing it, digging through the kernel code, it's out of my case. I figured out what seems to be that serial 5 and USB are both mixed to the Pogo pins. So it's these. That's all I have to see. But there were four, five pins on the side. Yeah, that was kind of useless. Five pins on the side. And it's the type of thing you'd attach an external keyboard accessory to. And so after double checking that my AB update was there and it's working, I made the plunge and edited the device tree on the host and SSH into the device and I synced it across. But I edited the device tree to change the U-Out to U-Out 5, rebooted and saw what happened. So luckily the device booted and I could see U-Out. So you see here in this picture, it's a little hard to see, but these are the five U-Out Pogo pins, sorry, five Pogo pins. And I've got connectors on them and I've got debug probes connected to two of them to get ground and TX out. So there's no U-Out in, so I can't send commands in, but I can receive messages from the remarkable and I can get the entire boot log from the kernel at this point. So next step was to flash the kernel. I couldn't spend the entire time building the kernel on my host, r-syncing it across, rebooting, seeing what happened, and then if that did work, that's great, but if it didn't work, hoping that the AB update still worked this time. That seemed too slow, too tedious. It was going to probably damage the flash with all these writes and wasn't really a workable solution. So luckily, there are other people in the community have figured out there is a recovery access, recovery mode. So using a resistor on two of the USB-C pins, you could short them and that would tell you, boot, to jump into the wrong download mode. And so by doing that, you could then use UU3U, which is a standard IMX tool to download recovery images. So this meant in this picture, so I have my Pogo pins like I had before, which plug into my computer and I can use my USB and then from my computer to download images with the same two debug probes connected to them to get serial out as well, while also using USB-C, extension cable, to break out two of the USB-C to then short a resistor across two of them. So with this setup, some sticky tape and a little bit of luck in holding the connectors, I could download images too remarkable, UGroot, Kernel, RooTFS, and Vystre, reboot and see the entire log. So now that I can boot a kernel and build myself, I want to do two things. First, I want to get a more recent kind of known working state. So I took the latest 5.4 NXP kernel, which is the latest kernel NXP I've released at this point. It was in a 5.10. And started porting the remarkable two features onto it. So this was about the 300 patches that I set originally. So I didn't port all, I kind of squashed them together into chunks. So they weren't ready to upstream, but they were kind of subdivided. So for example, I had like drivers input was one, add drivers input, add drivers input, top screen, things like that. So they're big chunks, but I think I got about down to about 30 patches at this point. And this worked. I could actually boot the remarkable, not just boot, I had it kind of up and running. This is not in the picture, I'll come to the picture a little bit. But at this point, it was actually working enough that after a few changes, I swapped across to using this full time. So I downloaded this onto the device, replaced the boot images with my own. And this is what I was doing for day to day use. So this is great. I had a nice working state. Instead of comparing against the 5.14, sorry, the 4.14 kernel, I was comparing against the 5.4 kernel now for my future work, which is much more recent at the time, I think, Linux would have been 5.12 or something, 5.13. So not that recent, but still better than 4.14. This is actually the first 5.4 kernel to run remarkable. The company itself updated later. But we are so yet the community was first to run the 5.4 kernel, at least publicly. And the 5.4 kernel used still by the human is still newer than the one released by the company. So that's all interesting. So the next step was mainline. So now that I had the NXP kernel working, I wanted to get mainline working. So I thought, how hard can it really be, right? Someone must have upstreamed all these NXP patches at this point. So it should just work. I mean, obviously not the remarkable part, but I thought that everything else would be okay. So I sat down, looked at some of the other IonX boards and kind of based a device tree up there with the minimum I could get away with, just basically you are nothing else. I also at this point thought, you know, I don't want to do, I don't want to boot this up with my, and use my home, my RoutaVest on my home directory on this thing. And something goes wrong, corrupts it and, and, you know, I can't get stuck back. So I thought also what I need is I need a RAM vest. I need something running in RAM that I can use instead. And so that's what open and better came in. So I knew open and better works. I use open and better all the time. It's a great, great distro tool then. And so I thought, you know, add a machine to that and use that. So there's a patch there. It's actually upstreamed as well. So the meta free scale third party machine comp. And that builds remarkable to kernel, U-boot, device tree and RoutaVest. And so I was using that. And with that, I did manage to boot just mainline. So mainline was just a device tree on top. I had UR and nothing else, no screen, no input, no nothing, but not, not, not even any Wi-Fi, but it was working. And that started the process of, of mainlining. So those patches were sent upstream, have since been merged. And that was kind of the first step. So finally in this, this bring up process, I thought, I need Wi-Fi. So like I mentioned earlier, there's no TA, there's no RX to the device. So I can't send a command to it. So I thought, well, you know, if, if once I get to use this space, what if I want to, you know, test something, I want to check something, I'm going to need Wi-Fi SSH at this point. So that's not too bad because the Wi-Fi driver was already upstream. So it's in the command line kernel. Okay, easy, right? I thought that would be the easy one. So the actual patch, five lines and device free patches to enable Wi-Fi and good to go. Unfortunately, this is probably one of the longest processes in the whole thing. So at first it wouldn't work because I didn't have the right firmware to the firmware version at the remarkable binary ships with an older one than what the kernel expects. So I tried to upstream that firmware to, and this is the same firmware actually the Raspberry Pi and the Raspberry Pi is used. So I tried to upstream that to the kernel firmware because I could get into my open embedded root of s. But that didn't go too well. Some license testing issues there. So eventually I just gave up and thought, you know what, I'm just going to boot and use the root of s on the device. How bad can it really be? Luckily, it wasn't that bad. But this then meant I compiled everything without modules. So I just put them the Wi-Fi driver into the kernel. That didn't work because that meant it tried to load it before the root of s was available and couldn't find the firmware. So eventually I figured that out. That's for people who do kernel development a lot. That's kind of obvious. But for people who don't, I don't have that much experience with Linux development. It took me a little bit to figure it out. So once I had that running, that was okay. So I could just copy modules across. I thought, you know, I'll sync them across. We're in the working setup and then I can reboot. And it worked a little bit, but it was really hidden next. I would add a debug statement and would just not work. And then I would remove it and it wouldn't work. And then add it and it would work and it couldn't figure it out what was going on. Eventually I realized the problem is that I was running the 5.4 custom kernel at this point and it doesn't have shut down support. I don't know why, but that's one of the things I never got working. So I can't turn it off. So I just hold the power button and force it off. And when I did that, it wouldn't always sync or flush everything to the EMMC. And so depending on the state, the kernel modules wouldn't work. And so eventually I figured out I have to make sure I run sync and flush everything to the internal EMMC before I power it off and reboot. And so eventually when I figured that out, combined with the different firmware versions and building the kernel modules, and actually I needed some device, some K-config changes, I had Wi-Fi working reliably and set my 5 line patch so I think I had run 2 months ago at this point or something. So that was a little embarrassing. So the Wacom. So I've got it here actually. So the Wacom style works kind of in real time. It's a bit hard to see, but let's see how this goes. So you can see it kind of draws as I draw it. So that's the latest, the 5.16.0. That's the patches I ended up with. So you can see here on the right-hand side of the screen, the patches from law. So this is the version 10 of my 12 patch series to add Wacom support. So this is a bit of a slog to get this going, but eventually at this point there were engineers from Wacom who were interested in seeing it merged as well. And this was all to add with the driver's touch screen, sorry, driver's input touch screen. There's a Wacom underscore s-grade c in there and that was this one. That was adding, improving the support there. And this is what the vendor did, so the remarkable guy did. So I thought that's the best place to be. Version 10 was finally nacked after all this time. So what happened in version 10 is the maintainers realized that the new Wacom's actually have full HID support. HID's a generic input interface, kind of pioneering input-screen Microsoft that really simplifies input. And the new Wacom ones support this. And as soon as they realized that, rightly so, they were rightly so. But they nacked my patches and said no, you have to do that. So at this point, it's deleted everything that I worked on and came back to it and ended up with basically two patches, one mostly just device tree, to add Wacom support to the remarkable. So it was already in there. I could just utilize all of the HID input framework. So in future, it's probably worth thinking about if there's generic frameworks you can use before you go run now and write your special custom driver. So the only problem with that was that the remarkable input is actually kind of rotated around 180 degrees. So the X and Y inputs had to be inverted. And in the Wacom driver, that's easy, you know, you have control over everything. But in the generic HID input, it's a little more difficult. So it took me a while to understand how the events worked and where this should actually go and what value we should be negating. But after that, it's still also difficult to modify the stuff because it affects lots of other things. So anyone who does have experience with drivers, you know, drivers input touchscreen knows there's always touch invert X, Y, device tree properties. So throw that in the dump. But like I said, though, here wasn't it wasn't implemented. So I ended up implementing that. So now we have support for that in in HID input and the ispread C HID. And that's what we use in the remarkable. So we can now flip them and negate the X and Y values, which works perfectly. As you just saw, you can now draw around and exactly matches where I expected to match on the remarkable. Here's a picture I took of FOSDEM logo on the remarkable at home. So like I kind of mentioned earlier, eating some pretty cool technology, you know, only updates when it only uses power when you when you update. And but one of the things about it is actually quite complex to use. And so because of this, you see most products use an IMX because it has a hard way in built called EPDC, which is that eating display controller, basically. And this is what kindles and cobers and all sorts of things use. And the remarkable uses it to the remarkable, though, is different in that they don't actually use this hardware. They instead just connect the screen directly to the LCD controller. And what that means is there's no way for the hardware to manage the waveforms or any of the other complex stuff related to eating. So I'm not going to too much detail about what that is. There's someone else that follows them dev room is going to talk about about driving eating displays. I'll leave that up to them. But basically, for the remarkable, it means there's a complex proprietary user space app that does manage all the waveforms. And it writes that data just to a frame buffer device. And the kernel, it has an MxS, FB, which is just a frame buffer by device for the IMAX. And that's what it uses. So this works great in the vendor panel. And looking at it, there even was that this device had been upstreamed and was upstreamed in the mainline kernel. Great. So they had us have to pull over some vendor patches and be good to go. But then I realized what actually happened is the vent, this frame buffer device, this MxS, FB, has been removed from mainline. So they had it and they removed it because it's been replaced with a DRM, a DRM driver. And that's what everyone's using upstream now. Unfortunately, the proprietary user space software doesn't work with the DRM driver. And I have no way to change it. So we're stuck using the frame buffer device. So no worries, we can just emulate the frame buffer device, right? So I tried that, have to write this down on my notes. But as a FBI, I put the screen info Ioptal that the proprietary user space uses. And that's trying to change some of the screen resolutions. And that's not supported in emulated frame buffer device. So we can't actually use this. So what we have is we have this new great DRM driver that should work. It doesn't because the proprietary user space doesn't support it. So in the end, I have to just backport, basically revert this MxS FB driver. And that's what I'm using today. And that's what I'm using my demo just then. And that's what I'm currently using. But that's not really main lineable. So there's two options. So obviously, we could extend frame buffer emulation to improve that to then support this new Ioptal and changing the screen resolutions. The other option is just remove the proprietary user space software. And there's an effort on going, not by me, but by others in the community to reverse engineer the e-ainting driving system and run an open source one. So this would be great because then we'd have a fully open source stack. And we could also then add DRM support to this software. So we don't even need frame buffer. We can just use a DRM directly. And so I'm hoping that's what's going to happen. And I have more details in the links at the end of the presentation too. So as you kind of go along, the way I tested things has changed. So initially, I would have that set up from the start with the upside down on the chair, and I would flash a kernel, make sure it booted. And then, for example, testing the wake home, I'd pick it up and draw a picture and make sure it worked. And then when I was done, I'd turn it off and reboot back to my custom 5.4 kernel that I kind of started with and go about using that. So eventually, at this point now, with about the 5.16, 5.17 kernel and extra patches on top, not just mainline, but at a point where we're going to daily drive it. So I'm now using this thing daily with the mainline, the mainline-inch kernel. And that's when you start to notice lots of other bugs that you probably wouldn't notice just in this quick testing turnaround. And so one of them is that I would put the device to sleep, walk away, and come back later. And it would be out of batteries if it was completely flat. But the weird thing with eating is that when the battery just dies or the machine just turns off, you see what it was lost doing because it doesn't clear the display. And so I could see that it was on the lock screen. And so first, it's really confusing because you try and touch it and it doesn't react and it eventually takes you a while to realize it's turned off. But once you get around that, I realized that it was, I thought it must be waking up, right, and putting it to sleep and something's happening, some interrupt somewhere, probably, you know, the touch screen is waking up and it's draining the batteries because I have sleep disabled. So once it wakes up, I'm going to go back to sleep unless you press the button. So okay, so that must be it. So eventually I was watching it one day and I realized what's actually happened, it's not what's happening, what's actually happening is after it goes to sleep for a few minutes, it would reboot, the whole thing would turn off and turn back on again. And then when it turns on, it gets to the lock screen and just sits there because like I said, I have sleep disabled, automatic sleep, and it will just burn through the battery and turn off. So okay, so it's reboot, that's a strange one. So I was like, okay, what could be maybe is a system D is crashing and it's causing a reboot that's possible, I guess, because I'm running their root of S and with my kernel. Or the other option is it's a watchdog timer and there's a watchdog rebooting. So I had a quick look at some of the logs and didn't really think it was the first one, system D one. So I thought, you know, it's probably a watchdog issue. And so I started digging into the kernel. And yeah, okay, there's a watchdog timer support, there's a driver for it and everything. Okay, when it goes to suspend, it sets the timer for two minutes. That's the maximum you can do about two minutes. That's the maximum you can do on 9 and X. And I looked at it and I was like, that's about when it turns up, it waits about two minutes in every boot. Okay, so it must be a watchdog timer. The watchdog timer is kicking in. But this is a mainline device, right? I didn't write this, this is already there. So does that mean everyone else has this device or every other IMX device reboots two minutes after going to sleep? That didn't seem right. So it's really through the specs, trying to figure out what's going on. And eventually I realized that although the watchdog timer cannot be disabled by software, so once you enable it, it can't be disabled. It does get disabled by hardware when you go into low power mode. So when you go into low power mode, the hardware will automatically kind of suspend the timer and you won't reboot. So why isn't that happening now? And so I dug into a little bit more and realized that there's no low power mode for the IMX7 or the IMX7D and not in mainline. So the vendor kernel works fine because they have low power mode implemented and they use that. So they don't have to worry about it. I guess other IMX devices are using low power mode too or don't suspend. And so they don't have to worry about it. But in this case, it don't have low power and I'm suspending. So eventually the watchdog times out and reboots the system. So I thought, okay, there must be a way for like a way to tell the kernel not to do that. I wanted to ping the watchdog every two minutes, looking at it for less than two minutes to keep it, make sure the watchdog time doesn't reboot on me. But there is a way. The kernel can do that. In fact, in the IMX driver, we're specifically telling the kernel not to do that. We're saying don't ping on suspend. And so that's because everyone else goes into low power mode and they don't need to. So for them, it can be a way to see a battery to wake up to ping this watchdog timer that's being disabled. So there's now patches sent and maybe by the time you read this, that we merge, but patches are on the list to add watchdog timer ping on suspend support for certain devices like the remarkable two. So now we can ping the watchdog on suspend. So we're not going to go to sleep. So that's a type of example of like a debugging issue or hit, not just from plugging in and testing it at your desk. We actually start carrying around and using it and putting it to sleep and leaving it for long periods of time, those two minutes isn't that long, but yet leaving it and using it and daily driving it. So 5.17 kernel. So the 5.16 kernel was released just for recent, not a little few weeks ago, I guess a week or two ago before I record this. So the 5.17 kernel is a little bit off, but the 5.17 kernel will have full boot support like I've talked about, Wi-Fi support and the way you can touch screen support. So that's pretty cool. The patches on list for the Syllogy NFD. So that's the power regulator to control the touch screen, control the screen, the LCD support, the LCD driver support, the cypress touch screen, which is like the finger touch screen. Now the really cool thing about that actually is that the same touch screen used in the Pine note, which is another eReader announced with Pine64. So some people have actually reached out to me about getting that to work on the Pine note. So the drive is in a process of being upstream and it needs a few changes for Pine note support, but it has been tested on the remarkable end of one of the co-votes and works on both of those. And the IMX Watchdog ping support that I just talked about is also has patches on list. So the next steps, so the current patches on list merged, get a toll tech package merged. So that's the remarkable package manager system. So I have actually Linux 5.16 support as a package in there. So I need someone to test that and merge that. So if anyone has a remarkable two-year-longster test, that's a great place to go. Get low power mode working. I've kind of talked about that a little bit. Fix up some paper cut bugs. There's a few kind of smaller issues. And reverse engineer the EEN driver like I talked about earlier. There's a link to the waived project. If anyone's interested, that'd be a great effort. And so that's it from me. I'm right at the end of my time. So I'm going to drop off now and I'll be around to insert any.
talk about my experiences in upstreaming and the support for the E Ink tablet to the Linux kernel. software. So it ships to the 4.14 kernel, which on that. And there's also no lockdown on the boot there's about 300 patches on top of the NXP vendor kernel from Remarkable. I also found that first I needed serial access. In order to do that was a real downer. This is a large blocker. then from my computer to download images with the kernel to run remarkable. The company itself was working. And that started the process of, of screen, the patches from law. So this is the upstreamed in the mainline kernel. Great. So they