The next generation of Matrix interfaces
Hunt, Will
Matrix has already got the solid framework of many bridge and bot implementations, but has always been missing that polish to make the bridges more accessible. In this talk, Will will explain how we're going to build our bridges with an all-new interactive interface and replacing old bot command interfaces with widget based interfaces.
 you Professor Hello and welcome to this talk, another matrix talk about the next generation of matrix interfaces. I will or half shot you might see me around I tend to work on the bridgey side of things and today we're working on a new thing interfaces relating to bridges and integrations. So this presentation is going to cover how bridge interfaces work today and how integrations work in such a way that users can interact with them so bot commands or other sort of forms of clicky button type stuff. How are we going to try and change things? Preface this talk is very much about experimenting and trying something new. This is an officially sanctioned matrix thing we're going to do tomorrow but this is certainly something that I've been experimenting with the last few weeks and I'm hoping this will change the way we do integrations going forwards. Now finally a little bit on what's new in bridge land and just a little bit of news to catch up on really. So first off let's talk about bridges. I won't get too much detail because I'm pretty certain most of the audits have a vague idea of bridges or overly building bridges usually one or two camps. Closely don't know bridges basically are a gluing method mechanism for connecting remote services like telegram or RFC into matrix. Typically they tend to exist on next-door home server a bit like this so you see here we've got our clients we've got our home servers and we've got application services. The application services is where the bridge is you lives it's connected you to at least one home server so it sort of works in a meshy style system. See here this is our example network of a bunch of different siloed services you've got your telegram, RFC, Slack and Gitter and so on and they all sort of exist separate from each other. Some of them might connect to each other there's not a general meshing system and of course this is what matrix provides so everyone's fairly aware that at this point matrix is the glue between them. I mean if you've come to foster and through RFC then you already notice because you're talking through RFC to matrix or Slack perhaps but yeah generally speaking this is this is our bread and butter of how we do stuff. So this is bridges across section example of how we build a bridge today how we do build bridges, an example of a Twilio bridge this is actually one I use in my previous talk at Comcom where I built a bridge about 100 lines of code to prove that you could basically build an SMS bridge with very little experience and you know just an hour of your time really. In this case I'm just showing off that we've got the client on the far end which is 10-bit users used to interface with the home server and the bridge and so on and then the bridge sits in the middle there talking between home server and the remote API. What's interesting here basically is that the user never directly interfaces through a bridge there's always a home server in the way and sort of contextualize this is some of the struggles that we're going to see later with why bridges sometimes are very difficult to control. Now we can talk a bit about interfaces. So interfaces in what we're using here is to mean how do users interact with services that aren't other humans I guess to put it plainly. If you have a bot in a chat or a bridge or some other service where the user needs to send a command or do a thing how do they do it today. Now they do today it looks a bit like this we have commands basically I probably build up the majority of our communication team these services is all through command line interfaces or whatever you prefer to call them. They've been with us for a very long time I think for instance the old days of matrix until mostly up till this day we still use commands to control bridges. There's a few cases we might have special metaphors like you can join you don't have to send apart a join a IOC network you can use aliases to join them but by and large you would do a thing particularly complicated like set your nixer password or if you want to do something like join a unbridged room or something particularly clever that will have to be done through commands which kind of sucks because command interfaces are not great. It's pretty painful honestly to use these things it's going through a bit more you can see that they're pretty intuitive for new users who have never used a command interface before. Most people today that are joining element are probably not used to using sort of TUI based stuff especially the cam3 one the mobile apps or you know how looking through Facebook or something they tend to be doing using some clicky button stuff or invites or QR codes and that's perfectly fine we want to support these users much more support the more hardcore users out there. The problem we have is as soon as you want to escape the bubble of the simple matrix communication chat or the room directory jumping into rooms you can get more complicated than that and you actually want to start using the more powerful features you'll suddenly hit with this wall of commands and I don't think it's what people want to see. Matrix clients are built by graphic interfaces you know a lot of our clients are quite clever and they look nice and it's great but then some of these issues use integration so it drops down to the floor of now you're just typing commands typing and help commands looking for wiki pages and that stuff just it ruins the experience I think to some degree. We shouldn't really expect matrix users to be typing bot commands this isn't to say that we should kill them off tomorrow this is to say there should not be the default this shouldn't be the way we're expecting users to use our stuff so how do we fix this? We could define a matrix spec to cover the things that bot commands cover today so we could add extra commands to do things like rolling a dice or we could do we could try and cover every possible use case for the very more extreme use cases out there but typically this doesn't tend to work because it increases the bloat of the spec so every little feature you want to add to maybe do your small little game on the side means we have to add to the spec which really means the client has to go and implement the feature in some form and their stuff this might be the best for open standards and ensuring that you know everybody's wrong on the same industry potentially but it has the effects that most clients probably into implementing it you'll have a sort of split grain between maybe element implementing this feature and then a bunch of clients not implementing it so it doesn't necessarily solve the problem for the small use cases certainly for the bigger things out there we should try to ensure a spec but for the small little extra features it might not be the super way to do it we should try and we could try and define a common language to define all the extra functionality so some sort of I don't know XML based language where you could define your forms and things and buttons and some sort of scripting language where you could just interpret the command and then send it back but we have that already this is basically widgets matrix widgets so people don't know which is actually are because they actually only end up in that so far not in most clients don't implement them but this is hopefully going to change at some point but widgets in the nutshell are HTML interfaces or HTML pages which you can insert into a matrix room they're completely sandboxed from the matrix of clients so they have an API that can talk through to a client but they don't find a way to have access to all the all the databases and bits and bulbs of the main client they're a separate entity and the idea is basically you can build your interfaces or hook in your existing website or something like that into your interface and does how it sit alongside your matrix room so if you had a collaborative document you want it on the side of your chat you can input that as a widget and we do that sometimes a YouTube video or something or how interface to control your matrix bot or something the other nice thing is it has a capacity negotiation system so if you want the ability to limit what the system can do is they can't read your events or it can't read your account data by default which is a very limited view to your account but if you want to grant find range control to your account that's a thing they support and actually it's a super powerful feature when you consider that you don't necessarily want to have full access to your account architecturally they look a bit like this you have a your client again bits and a lot of diagram we have a widget that sits sort of on top of his own widget API in an example we'll see a bit later there's a rest API which you you can have a rest API team or widget and the bridge and the widget also has access to some of the data you'd have on your matrix account which it grants through the widget API you can sort of see here it's not a direct connection to my myself which it can't directly start calling client server API can own stuff at least not out of the box you could potentially provide a access token to your widget but we try to make sure that all the calls to the home so go through the client to avoid sort of any potential dangerous actions but yeah typically this is how we would build a thing so next one I talk about is how do we invite users to a Dm over a bridge and this is a common problem we've got with bridges at the moment is bridges can't yet provide a mechanism to invite users who have never spoken on matrix we're perfectly so able to invite users who are already chatting on matrix so I can always invite someone from my RSE conversation to a chat at Dm that's fine but the user has never used the bridge before how does matrix show a UI for users which have never set up a profile never really interacted you'd have to have some sort of directory system this can be achieved with other mechanisms but one way it's really easy to achieve is to simply a widget in your room something where you can just type in the name the bridge will search for them and in return they was all and this is actually what we're going to demo just now so if I switch over to my lovely slack demo here you can see here that I have got a matrix cancels up on element this has got a few things on it hookshot bridge, sherry and bridge and slack bridge you can see here we've already got a bunch of things in here crucially we have this bridge control widget I just want to quickly show off the widget itself not the actual widget not the widgets UI but actually the content of the widget you can see here they've got a zoom in a bit that's actually visible you've got a UK harsh or bridge room subscription Spain which basically says hello this widget provides a feature for your bridge in this case it provides an invite widget feature and I'm going to see how that actually looks so I go to start chat and I'm a client look for people you see you've got the usual stuff but interestingly you've got this bridges section with a hookshot settings room which corresponds to this guy here but also a slack bridge room which corresponds to our slack bridge so I click through to this and I actually granted permissions to again this is our permission system granting permissions to verify that my user ID we can hit here and then suddenly presented with this lovely dialogue box I will admit my design credentials and not as good as my bridging and stuff but generally speaking this is a simple simple UI and houses such users on slack so what this is actually doing is rendering a very element itself is rendering a very generic window which the bridge them fills in with some UI so this bit here is all running on top of the bridge and this outer bit is element so if I don't search for foster you can see here there's this user here could will foster never spoke to him for my account but now if I click through now and click this guy and then type here and go to here I've now got invited I've now sorted the end of this person and then if I don't send a message saying hello and hopefully this person's actually respond to me we should find out here we are upset hello to my back and if I don't say hello back just to prove the fact that this is definitely a live demo you can see here that's a message back so proof here is you basically can then look for people through this lovely system and it also works I mean I could also prove that it's fine to me myself so that's cool if I did well does this because in our particular demo for the slack bridge it actually searches both your name and your display and it's it can find people through multiple directory keys if you were and so that's pretty cool so that's our demo for how to do widgets and a very basic demo how widgets actually work in a bridge context of course you can do way more than that we can have rich settings rooms and things like that or we could have images appearing in the rooms or one of the things I've always loved to have the idea of it's a side winding room widget on one side of the room on the other side so you could see news or some sort of like extra data that your room doesn't be able to provide on its own but this is like sort of the idea of where you could extend this further if you wanted to write multiple sort of invite types so let's talk about the forex which you've seen me leak slightly just now so one of things we can still that still are missing is obviously that not all clients support widgets so we're trying to think about how we can do full backs for this one idea is the idea that since the widgets just calling rest API's we just expose those rest API's and authentication mechanism and so the client itself could call these and this would be good for the case where you are a client that couldn't doesn't we want to bundle a whole web rendering engine but would love to be a support add under the RSC bridges the widget API and so what you could do is also inspect this open API spec and then potentially build out an interface on top of that and that's one idea exploring some sort of limited form of also form building we want to be very careful there and make sure that you still have the ability to do nuanced things and more than just a common basic form we don't really want to start specking a whole forms in matrix thing given that's a solved problem elsewhere and of course the one thing we want to ensure we still have is bot commands as a fullback and it's in a way that in my own Linux you'll have you know fullback commands so pretty much if you do an UI same on the GUI the same reason we want to make sure the matrix still has that fullback for people who don't want to present a UI which is still a very useful thing to have but otherwise yeah we're hoping this will be a way in which we can sort of build much richer integrations into existing clients so next steps how do we take this further so we've got some MSCs on the way as you would expect with most of these things there are two MSCs currently out there and how we can expose user IDs from clients from widgets to clients so you can start in the invite or start a DM I've got one of them 3662 which is very particularly about user IDs in texture how you'd it convey intense to the client about what you plan to do with them there's already an existing MSC 2931 which handles room IDs and events too so it might be one of those will probably end up winning the feature end up in one of these one of these ways and there's also the widget spec itself which is in the draft also massive huge congrats to Travis for writing a massive massive document about how you do widgets and it's like pretty complete but just not landed in the spec yet for what it's worth most of the widget stuff is fairly stable at this point in terms of actual usage a lot of widgets already in use by element in various places so we're pretty confident of the actual technology itself we also need to land the bridge components for this demo with what I demoed there was a bunch of stuff that's in branches that most of it's in a sort of draft state and the next thing I'd like to do is I'll take this further and have it land as part of the core makeshift source bridge SDK some work and start using it and I'd also like to land the other web stuff there let's just saw there again most of it was branches but perfectly reasonable one of things I like to do them web implementation in particular is have more settings view so you can see which bridges you subscribe to and also how to sort of control which bits and you are they sharpen we like to develop the full back solution talked about earlier so it's it's old clients data left behind a crucial part of this whole thing is that we don't lose clients because only one thing wasn't trying to build a better you why is you end up killing off a whole bunch of your existing user base by end up having no one who can use your stuff so the only way we'd ever a progressive as widget stuff is we kept access to all the existing commands and made it still as easy use as possible so that would be one of the challenges we'd like to look at first and of course if all of this stuff happens we'd like to make sure that the IC bridge gains it and then the slack bridge obviously improves its support and so forth we'd like to make sure eventually all the bridges become as ergonomic as this can be so yeah finally fine thing we'll talk about is basically what's been happening in bridge land already so there's a few things out there both in what we call out ticklers and to end to bridge encryption is now formally landed as inside synapse and the botus decay the setup stuff is very very close there's a few mergers to happen for it's like in the stable but I've played a bit and it's absolutely brilliant and so finally we've solved this problem where bridges can now participate in encrypted rooms without the overhead of having to call a sync request to get things like device messages down and it's a little bit technical but while the existing forms have had with end to bridge encryption is that device to pull down information about encryption you had to hit sync APIs on the server which was a polling action and the polling action across different users meant that it was a bit slow and all the bridge bot had to be in lots of rooms which you didn't need to be in and the new stuff is all happens down the transaction API which is a push technology which means that the hope so does sends data to the app service which is meant to be a bit better for the whole performance footprint so that's looking pretty good the botus decay itself has landed all the components inside its beta branch and I've been using it already it's actually powered by the message rusts SDK under the hood which is pretty cool in itself but yeah one of things to look out for is the fact that maybe this year most of our bridges will gain into bridge encryption by default which is just really cool the fact that we don't have to worry about whether a room's encrypted or not it's a bridge to it and of course one the downsides to end to bridge encryption is that and it's not fully encrypted but we're hoping at some point that encryption becomes property to weave these various networks if they all do not technology so we'll see otherwise that's my talk I'm hoping there'll be some questions but basically that covers everything I plan to talk about thank you
 you  Professor  Hello and welcome to this talk, another matrix talk about the next generation of  matrix interfaces. I will or half shot you might see me around I tend to work on  the bridgey side of things and today we're working on a new thing interfaces  relating to bridges and integrations. So this presentation is going to cover how  bridge interfaces work today and how integrations work in such a way that  users can interact with them so bot commands or other sort of forms of  clicky button type stuff. How are we going to try and change things? Preface  this talk is very much about experimenting and trying something new.  This is an officially sanctioned matrix thing we're going to do tomorrow but  this is certainly something that I've been experimenting with the last few  weeks and I'm hoping this will change the way we do integrations going forwards.  Now finally a little bit on what's new in bridge land and just a little bit of  news to catch up on really. So first off let's talk about bridges. I won't get  too much detail because I'm pretty certain most of the audits have a  vague idea of bridges or overly building bridges usually one or two camps.  Closely don't know bridges basically are a gluing method mechanism for  connecting remote services like telegram or RFC into matrix. Typically they tend  to exist on next-door home server a bit like this so you see here we've got our  clients we've got our home servers and we've got application services. The  application services is where the bridge is you lives it's connected you to at  least one home server so it sort of works in a meshy style system. See here  this is our example network of a bunch of different siloed services you've got  your telegram, RFC, Slack and Gitter and so on and they all sort of exist separate  from each other. Some of them might connect to each other there's not a  general meshing system and of course this is what matrix provides so everyone's  fairly aware that at this point matrix is the glue between them. I mean if you've  come to foster and through RFC then you already notice because you're talking  through RFC to matrix or Slack perhaps but yeah generally speaking this is  this is our bread and butter of how we do stuff. So this is bridges across section  example of how we build a bridge today how we do build bridges, an example of a  Twilio bridge this is actually one I use in my previous talk at Comcom where I  built a bridge about 100 lines of code to prove that you could basically build an  SMS bridge with very little experience and you know just an hour of your time  really. In this case I'm just showing off that we've got the client on the far end  which is 10-bit users used to interface with the home server and the bridge and  so on and then the bridge sits in the middle there talking between home server  and the remote API. What's interesting here basically is that the user never  directly interfaces through a bridge there's always a home server in the way  and sort of contextualize this is some of the struggles that we're going to see  later with why bridges sometimes are very difficult to control. Now we can  talk a bit about interfaces. So interfaces in what we're using here is to  mean how do users interact with services that aren't other humans I guess to put  it plainly. If you have a bot in a chat or a bridge or some other service where  the user needs to send a command or do a thing how do they do it today. Now they do  today it looks a bit like this we have commands basically I probably build up  the majority of our communication team these services is all through  command line interfaces or whatever you prefer to call them. They've been with us  for a very long time I think for instance the old days of matrix until mostly up  till this day we still use commands to control bridges. There's a few cases we  might have special metaphors like you can join you don't have to send apart a  join a IOC network you can use aliases to join them but by and large you would do  a thing particularly complicated like set your nixer password or if you want to  do something like join a unbridged room or something particularly clever that  will have to be done through commands which kind of sucks because command  interfaces are not great. It's pretty painful honestly to use these things it's  going through a bit more you can see that they're pretty intuitive for new  users who have never used a command interface before. Most people today that  are joining element are probably not used to using sort of TUI based stuff  especially the cam3 one the mobile apps or you know how looking through Facebook  or something they tend to be doing using some clicky button stuff or invites or  QR codes and that's perfectly fine we want to support these users much more  support the more hardcore users out there. The problem we have is as soon as  you want to escape the bubble of the simple matrix communication chat or the  room directory jumping into rooms you can get more complicated than that and  you actually want to start using the more powerful features you'll suddenly hit  with this wall of commands and I don't think it's what people want to see.  Matrix clients are built by graphic interfaces you know a lot of our clients  are quite clever and they look nice and it's great but then some of these issues  use integration so it drops down to the floor of now you're just typing commands  typing and help commands looking for wiki pages and that stuff just it ruins  the experience I think to some degree. We shouldn't really expect matrix users  to be typing bot commands this isn't to say that we should kill them off tomorrow  this is to say there should not be the default this shouldn't be the way we're  expecting users to use our stuff so how do we fix this? We could define a matrix  spec to cover the things that bot commands cover today so we could add  extra commands to do things like rolling a dice or we could do we could try and  cover every possible use case for the very more extreme use cases out there  but typically this doesn't tend to work because it increases the bloat of the  spec so every little feature you want to add to maybe do your small little game  on the side means we have to add to the spec which really means the client has to  go and implement the feature in some form and their stuff this might be the best  for open standards and ensuring that you know everybody's wrong on the same  industry potentially but it has the effects that most clients probably  into implementing it you'll have a sort of split grain between maybe element  implementing this feature and then a bunch of clients not implementing it so  it doesn't necessarily solve the problem for the small use cases certainly for  the bigger things out there we should try to ensure a spec but for the small  little extra features it might not be the super way to do it we should try and  we could try and define a common language to define all the extra  functionality so some sort of I don't know XML based language where you could  define your forms and things and buttons and some sort of scripting language  where you could just interpret the command and then send it back but we  have that already this is basically widgets matrix widgets so people don't  know which is actually are because they actually only end up in that so far not  in most clients don't implement them but this is hopefully going to change at  some point but widgets in the nutshell are HTML interfaces or HTML pages which  you can insert into a matrix room they're completely sandboxed from the matrix  of clients so they have an API that can talk through to a client but they don't  find a way to have access to all the all the databases and bits and bulbs of the  main client they're a separate entity and the idea is basically you can build  your interfaces or hook in your existing website or something like that into  your interface and does how it sit alongside your matrix room so if you  had a collaborative document you want it on the side of your chat you can input  that as a widget and we do that sometimes a YouTube video or something or  how interface to control your matrix bot or something the other nice thing is it  has a capacity negotiation system so if you want the ability to limit what the  system can do is they can't read your events or it can't read your account  data by default which is a very limited view to your account but if you want to  grant find range control to your account that's a thing they support and  actually it's a super powerful feature when you consider that you don't  necessarily want to have full access to your account  architecturally they look a bit like this you have a your client again bits and  a lot of diagram we have a widget that sits sort of on top of his own widget  API in an example we'll see a bit later there's a rest API which you you can  have a rest API team or widget and the bridge and the widget also has access to  some of the data you'd have on your matrix account which it grants through  the widget API you can sort of see here it's not a direct connection to my  myself which it can't directly start calling client server API can own  stuff at least not out of the box you could potentially provide a access  token to your widget but we try to make sure that all the calls to the home so  go through the client to avoid sort of any potential dangerous actions but  yeah typically this is how we would build a thing so next one I talk about  is how do we invite users to a Dm over a bridge and this is a common problem  we've got with bridges at the moment is bridges can't yet provide a mechanism to  invite users who have never spoken on matrix we're perfectly so able to invite  users who are already chatting on matrix so I can always invite someone from my  RSE conversation to a chat at Dm that's fine but the user has never used the  bridge before how does matrix show a UI for users which have never set up a  profile never really interacted you'd have to have some sort of directory  system this can be achieved with other mechanisms but one way it's really easy  to achieve is to simply a widget in your room something where you can just type  in the name the bridge will search for them and in return they was all and this  is actually what we're going to demo just now so if I switch over to my lovely  slack demo here you can see here that I have got a matrix cancels up on element  this has got a few things on it hookshot bridge, sherry and bridge and slack bridge  you can see here we've already got a bunch of things in here crucially we have  this bridge control widget I just want to quickly show off the widget itself not  the actual widget not the widgets UI but actually the content of the widget you  can see here they've got a zoom in a bit that's actually visible you've got a UK  harsh or bridge room subscription Spain which basically says hello this widget  provides a feature for your bridge in this case it provides an invite widget  feature and I'm going to see how that actually looks so I go to start chat  and I'm a client look for people you see you've got the usual stuff but  interestingly you've got this bridges section with a hookshot settings room  which corresponds to this guy here but also a slack bridge room which  corresponds to our slack bridge so I click through to this and I actually  granted permissions to again this is our permission system granting  permissions to verify that my user ID we can hit here and then suddenly  presented with this lovely dialogue box I will admit my design credentials and  not as good as my bridging and stuff but generally speaking this is a simple  simple UI and houses such users on slack so what this is actually doing is  rendering a very element itself is rendering a very generic window which  the bridge them fills in with some UI so this bit here is all running on top of  the bridge and this outer bit is element so if I don't search for foster you can  see here there's this user here could will foster never spoke to him for my  account but now if I click through now and click this guy and then type here and  go to here I've now got invited I've now sorted the end of this person and then  if I don't send a message saying hello and hopefully this person's actually  respond to me we should find out here we are upset hello to my back and if I  don't say hello back just to prove the fact that this is definitely a live demo  you can see here that's a message back so proof here is you basically can then  look for people through this lovely system and it also works I mean I could  also prove that it's fine to me myself so that's cool if I did well does this  because in our particular demo for the slack bridge it actually searches both  your name and your display and it's it can find people through multiple  directory keys if you were and so that's pretty cool so that's our demo for how to  do widgets and a very basic demo how widgets actually work in a bridge  context of course you can do way more than that we can have rich settings  rooms and things like that or we could have images appearing in the rooms or  one of the things I've always loved to have the idea of it's a side winding  room widget on one side of the room on the other side so you could see news or  some sort of like extra data that your room doesn't be able to provide on its  own but this is like sort of the idea of where you could extend this further if  you wanted to write multiple sort of invite types so let's talk about the  forex which you've seen me leak slightly just now so one of things we can still  that still are missing is obviously that not all clients support widgets so we're  trying to think about how we can do full backs for this one idea is the idea  that since the widgets just calling rest API's we just expose those rest API's  and authentication mechanism and so the client itself could call these and this  would be good for the case where you are a client that couldn't doesn't we want to  bundle a whole web rendering engine but would love to be a support add under the  RSC bridges the widget API and so what you could do is also inspect this open  API spec and then potentially build out an interface on top of that and that's  one idea exploring some sort of limited form of also form building we want to  be very careful there and make sure that you still have the ability to do nuanced  things and more than just a common basic form we don't really want to start  specking a whole forms in matrix thing given that's a solved problem elsewhere  and of course the one thing we want to ensure we still have is bot commands as  a fullback and it's in a way that in my own Linux you'll have you know fullback  commands so pretty much if you do an UI same on the GUI the same reason we want  to make sure the matrix still has that fullback for people who don't want to  present a UI which is still a very useful thing to have but otherwise yeah  we're hoping this will be a way in which we can sort of build much richer  integrations into existing clients so next steps how do we take this further  so we've got some MSCs on the way as you would expect with most of these things  there are two MSCs currently out there and how we can expose user IDs from  clients from widgets to clients so you can start in the invite or start a DM  I've got one of them 3662 which is very particularly about user IDs in texture  how you'd it convey intense to the client about what you plan to do with them  there's already an existing MSC 2931 which handles room IDs and events too so  it might be one of those will probably end up winning the feature end up in one  of these one of these ways and there's also the widget spec itself which is in  the draft also massive huge congrats to Travis for writing a massive massive  document about how you do widgets and it's like pretty complete but just not  landed in the spec yet for what it's worth most of the widget stuff is fairly  stable at this point in terms of actual usage a lot of widgets already in use  by element in various places so we're pretty confident of the actual technology  itself we also need to land the bridge components for this demo with what I  demoed there was a bunch of stuff that's in branches that most of it's in a  sort of draft state and the next thing I'd like to do is I'll take this further  and have it land as part of the core makeshift source bridge SDK some  work and start using it and I'd also like to land the other web stuff there  let's just saw there again most of it was branches but perfectly reasonable  one of things I like to do them web implementation in particular is have  more settings view so you can see which bridges you subscribe to and also how  to sort of control which bits and you are they sharpen we like to develop the  full back solution talked about earlier so it's it's old clients data left  behind a crucial part of this whole thing is that we don't lose clients  because only one thing wasn't trying to build a better you why is you end up  killing off a whole bunch of your existing user base by end up having no  one who can use your stuff so the only way we'd ever a progressive as widget  stuff is we kept access to all the existing commands and made it still as  easy use as possible so that would be one of the challenges we'd like to look at  first and of course if all of this stuff happens we'd like to make sure that the  IC bridge gains it and then the slack bridge obviously improves its support  and so forth we'd like to make sure eventually all the bridges become as  ergonomic as this can be so yeah finally fine thing we'll talk about is basically  what's been happening in bridge land already so there's a few things out  there both in what we call out ticklers and to end to bridge encryption is now  formally landed as inside synapse and the botus decay the setup stuff is very  very close there's a few mergers to happen for it's like in the stable but  I've played a bit and it's absolutely brilliant and so finally we've solved  this problem where bridges can now participate in encrypted rooms without  the overhead of having to call a sync request to get things like device  messages down and it's a little bit technical but while the existing forms  have had with end to bridge encryption is that device to pull down information  about encryption you had to hit sync APIs on the server which was a polling  action and the polling action across different users meant that it was a bit  slow and all the bridge bot had to be in lots of rooms which you didn't need to  be in and the new stuff is all happens down the transaction API which is a  push technology which means that the hope so does sends data to the app service  which is meant to be a bit better for the whole performance footprint so that's  looking pretty good the botus decay itself has landed all the components  inside its beta branch and I've been using it already it's actually powered by  the message rusts SDK under the hood which is pretty cool in itself but yeah  one of things to look out for is the fact that maybe this year most of our  bridges will gain into bridge encryption by default which is just really cool the  fact that we don't have to worry about whether a room's encrypted or not it's a  bridge to it and of course one the downsides to end to bridge encryption is  that and it's not fully encrypted but we're hoping at some point that encryption  becomes property to weave these various networks if they all do not technology so  we'll see otherwise that's my talk I'm hoping there'll be some questions but  basically that covers everything I plan to talk about thank you
matrix_next_gen_interfaces_0_3311.png matrix_next_gen_interfaces_3312_5699.png matrix_next_gen_interfaces_5700_5759.png matrix_next_gen_interfaces_5760_6191.png matrix_next_gen_interfaces_6192_11351.png matrix_next_gen_interfaces_11352_11435.png matrix_next_gen_interfaces_11436_15635.png matrix_next_gen_interfaces_15636_19295.png matrix_next_gen_interfaces_19296_19679.png matrix_next_gen_interfaces_19680_20291.png matrix_next_gen_interfaces_20292_23987.png matrix_next_gen_interfaces_23988_27299.png matrix_next_gen_interfaces_27300_28799.png
you Professor Hello and welcome to this talk, another matrix talk about the next generation of matrix interfaces. I will or half shot you might see me around I tend to work on the bridgey side of things and today we're working on a new thing interfaces relating to bridges and integrations. So this presentation is going to cover how bridge interfaces work today and how integrations work in such a way that users can interact with them so bot commands or other sort of forms of clicky button type stuff. How are we going to try and change things? Preface this talk is very much about experimenting and trying something new. This is an officially sanctioned matrix thing we're going to do tomorrow but this is certainly something that I've been experimenting with the last few weeks and I'm hoping this will change the way we do integrations going forwards. Now finally a little bit on what's new in bridge land and just a little bit of news to catch up on really. So first off let's talk about bridges. I won't get too much detail because I'm pretty certain most of the audits have a vague idea of bridges or overly building bridges usually one or two camps. Closely don't know bridges basically are a gluing method mechanism for connecting remote services like telegram or RFC into matrix. Typically they tend to exist on next-door home server a bit like this so you see here we've got our clients we've got our home servers and we've got application services. The application services is where the bridge is you lives it's connected you to at least one home server so it sort of works in a meshy style system. See here this is our example network of a bunch of different siloed services you've got your telegram, RFC, Slack and Gitter and so on and they all sort of exist separate from each other. Some of them might connect to each other there's not a general meshing system and of course this is what matrix provides so everyone's fairly aware that at this point matrix is the glue between them. I mean if you've come to foster and through RFC then you already notice because you're talking through RFC to matrix or Slack perhaps but yeah generally speaking this is this is our bread and butter of how we do stuff. So this is bridges across section example of how we build a bridge today how we do build bridges, an example of a Twilio bridge this is actually one I use in my previous talk at Comcom where I built a bridge about 100 lines of code to prove that you could basically build an SMS bridge with very little experience and you know just an hour of your time really. In this case I'm just showing off that we've got the client on the far end which is 10-bit users used to interface with the home server and the bridge and so on and then the bridge sits in the middle there talking between home server and the remote API. What's interesting here basically is that the user never directly interfaces through a bridge there's always a home server in the way and sort of contextualize this is some of the struggles that we're going to see later with why bridges sometimes are very difficult to control. Now we can talk a bit about interfaces. So interfaces in what we're using here is to mean how do users interact with services that aren't other humans I guess to put it plainly. If you have a bot in a chat or a bridge or some other service where the user needs to send a command or do a thing how do they do it today. Now they do today it looks a bit like this we have commands basically I probably build up the majority of our communication team these services is all through command line interfaces or whatever you prefer to call them. They've been with us for a very long time I think for instance the old days of matrix until mostly up till this day we still use commands to control bridges. There's a few cases we might have special metaphors like you can join you don't have to send apart a join a IOC network you can use aliases to join them but by and large you would do a thing particularly complicated like set your nixer password or if you want to do something like join a unbridged room or something particularly clever that will have to be done through commands which kind of sucks because command interfaces are not great. It's pretty painful honestly to use these things it's going through a bit more you can see that they're pretty intuitive for new users who have never used a command interface before. Most people today that are joining element are probably not used to using sort of TUI based stuff especially the cam3 one the mobile apps or you know how looking through Facebook or something they tend to be doing using some clicky button stuff or invites or QR codes and that's perfectly fine we want to support these users much more support the more hardcore users out there. The problem we have is as soon as you want to escape the bubble of the simple matrix communication chat or the room directory jumping into rooms you can get more complicated than that and you actually want to start using the more powerful features you'll suddenly hit with this wall of commands and I don't think it's what people want to see. Matrix clients are built by graphic interfaces you know a lot of our clients are quite clever and they look nice and it's great but then some of these issues use integration so it drops down to the floor of now you're just typing commands typing and help commands looking for wiki pages and that stuff just it ruins the experience I think to some degree. We shouldn't really expect matrix users to be typing bot commands this isn't to say that we should kill them off tomorrow this is to say there should not be the default this shouldn't be the way we're expecting users to use our stuff so how do we fix this? We could define a matrix spec to cover the things that bot commands cover today so we could add extra commands to do things like rolling a dice or we could do we could try and cover every possible use case for the very more extreme use cases out there but typically this doesn't tend to work because it increases the bloat of the spec so every little feature you want to add to maybe do your small little game on the side means we have to add to the spec which really means the client has to go and implement the feature in some form and their stuff this might be the best for open standards and ensuring that you know everybody's wrong on the same industry potentially but it has the effects that most clients probably into implementing it you'll have a sort of split grain between maybe element implementing this feature and then a bunch of clients not implementing it so it doesn't necessarily solve the problem for the small use cases certainly for the bigger things out there we should try to ensure a spec but for the small little extra features it might not be the super way to do it we should try and we could try and define a common language to define all the extra functionality so some sort of I don't know XML based language where you could define your forms and things and buttons and some sort of scripting language where you could just interpret the command and then send it back but we have that already this is basically widgets matrix widgets so people don't know which is actually are because they actually only end up in that so far not in most clients don't implement them but this is hopefully going to change at some point but widgets in the nutshell are HTML interfaces or HTML pages which you can insert into a matrix room they're completely sandboxed from the matrix of clients so they have an API that can talk through to a client but they don't find a way to have access to all the all the databases and bits and bulbs of the main client they're a separate entity and the idea is basically you can build your interfaces or hook in your existing website or something like that into your interface and does how it sit alongside your matrix room so if you had a collaborative document you want it on the side of your chat you can input that as a widget and we do that sometimes a YouTube video or something or how interface to control your matrix bot or something the other nice thing is it has a capacity negotiation system so if you want the ability to limit what the system can do is they can't read your events or it can't read your account data by default which is a very limited view to your account but if you want to grant find range control to your account that's a thing they support and actually it's a super powerful feature when you consider that you don't necessarily want to have full access to your account architecturally they look a bit like this you have a your client again bits and a lot of diagram we have a widget that sits sort of on top of his own widget API in an example we'll see a bit later there's a rest API which you you can have a rest API team or widget and the bridge and the widget also has access to some of the data you'd have on your matrix account which it grants through the widget API you can sort of see here it's not a direct connection to my myself which it can't directly start calling client server API can own stuff at least not out of the box you could potentially provide a access token to your widget but we try to make sure that all the calls to the home so go through the client to avoid sort of any potential dangerous actions but yeah typically this is how we would build a thing so next one I talk about is how do we invite users to a Dm over a bridge and this is a common problem we've got with bridges at the moment is bridges can't yet provide a mechanism to invite users who have never spoken on matrix we're perfectly so able to invite users who are already chatting on matrix so I can always invite someone from my RSE conversation to a chat at Dm that's fine but the user has never used the bridge before how does matrix show a UI for users which have never set up a profile never really interacted you'd have to have some sort of directory system this can be achieved with other mechanisms but one way it's really easy to achieve is to simply a widget in your room something where you can just type in the name the bridge will search for them and in return they was all and this is actually what we're going to demo just now so if I switch over to my lovely slack demo here you can see here that I have got a matrix cancels up on element this has got a few things on it hookshot bridge, sherry and bridge and slack bridge you can see here we've already got a bunch of things in here crucially we have this bridge control widget I just want to quickly show off the widget itself not the actual widget not the widgets UI but actually the content of the widget you can see here they've got a zoom in a bit that's actually visible you've got a UK harsh or bridge room subscription Spain which basically says hello this widget provides a feature for your bridge in this case it provides an invite widget feature and I'm going to see how that actually looks so I go to start chat and I'm a client look for people you see you've got the usual stuff but interestingly you've got this bridges section with a hookshot settings room which corresponds to this guy here but also a slack bridge room which corresponds to our slack bridge so I click through to this and I actually granted permissions to again this is our permission system granting permissions to verify that my user ID we can hit here and then suddenly presented with this lovely dialogue box I will admit my design credentials and not as good as my bridging and stuff but generally speaking this is a simple simple UI and houses such users on slack so what this is actually doing is rendering a very element itself is rendering a very generic window which the bridge them fills in with some UI so this bit here is all running on top of the bridge and this outer bit is element so if I don't search for foster you can see here there's this user here could will foster never spoke to him for my account but now if I click through now and click this guy and then type here and go to here I've now got invited I've now sorted the end of this person and then if I don't send a message saying hello and hopefully this person's actually respond to me we should find out here we are upset hello to my back and if I don't say hello back just to prove the fact that this is definitely a live demo you can see here that's a message back so proof here is you basically can then look for people through this lovely system and it also works I mean I could also prove that it's fine to me myself so that's cool if I did well does this because in our particular demo for the slack bridge it actually searches both your name and your display and it's it can find people through multiple directory keys if you were and so that's pretty cool so that's our demo for how to do widgets and a very basic demo how widgets actually work in a bridge context of course you can do way more than that we can have rich settings rooms and things like that or we could have images appearing in the rooms or one of the things I've always loved to have the idea of it's a side winding room widget on one side of the room on the other side so you could see news or some sort of like extra data that your room doesn't be able to provide on its own but this is like sort of the idea of where you could extend this further if you wanted to write multiple sort of invite types so let's talk about the forex which you've seen me leak slightly just now so one of things we can still that still are missing is obviously that not all clients support widgets so we're trying to think about how we can do full backs for this one idea is the idea that since the widgets just calling rest API's we just expose those rest API's and authentication mechanism and so the client itself could call these and this would be good for the case where you are a client that couldn't doesn't we want to bundle a whole web rendering engine but would love to be a support add under the RSC bridges the widget API and so what you could do is also inspect this open API spec and then potentially build out an interface on top of that and that's one idea exploring some sort of limited form of also form building we want to be very careful there and make sure that you still have the ability to do nuanced things and more than just a common basic form we don't really want to start specking a whole forms in matrix thing given that's a solved problem elsewhere and of course the one thing we want to ensure we still have is bot commands as a fullback and it's in a way that in my own Linux you'll have you know fullback commands so pretty much if you do an UI same on the GUI the same reason we want to make sure the matrix still has that fullback for people who don't want to present a UI which is still a very useful thing to have but otherwise yeah we're hoping this will be a way in which we can sort of build much richer integrations into existing clients so next steps how do we take this further so we've got some MSCs on the way as you would expect with most of these things there are two MSCs currently out there and how we can expose user IDs from clients from widgets to clients so you can start in the invite or start a DM I've got one of them 3662 which is very particularly about user IDs in texture how you'd it convey intense to the client about what you plan to do with them there's already an existing MSC 2931 which handles room IDs and events too so it might be one of those will probably end up winning the feature end up in one of these one of these ways and there's also the widget spec itself which is in the draft also massive huge congrats to Travis for writing a massive massive document about how you do widgets and it's like pretty complete but just not landed in the spec yet for what it's worth most of the widget stuff is fairly stable at this point in terms of actual usage a lot of widgets already in use by element in various places so we're pretty confident of the actual technology itself we also need to land the bridge components for this demo with what I demoed there was a bunch of stuff that's in branches that most of it's in a sort of draft state and the next thing I'd like to do is I'll take this further and have it land as part of the core makeshift source bridge SDK some work and start using it and I'd also like to land the other web stuff there let's just saw there again most of it was branches but perfectly reasonable one of things I like to do them web implementation in particular is have more settings view so you can see which bridges you subscribe to and also how to sort of control which bits and you are they sharpen we like to develop the full back solution talked about earlier so it's it's old clients data left behind a crucial part of this whole thing is that we don't lose clients because only one thing wasn't trying to build a better you why is you end up killing off a whole bunch of your existing user base by end up having no one who can use your stuff so the only way we'd ever a progressive as widget stuff is we kept access to all the existing commands and made it still as easy use as possible so that would be one of the challenges we'd like to look at first and of course if all of this stuff happens we'd like to make sure that the IC bridge gains it and then the slack bridge obviously improves its support and so forth we'd like to make sure eventually all the bridges become as ergonomic as this can be so yeah finally fine thing we'll talk about is basically what's been happening in bridge land already so there's a few things out there both in what we call out ticklers and to end to bridge encryption is now formally landed as inside synapse and the botus decay the setup stuff is very very close there's a few mergers to happen for it's like in the stable but I've played a bit and it's absolutely brilliant and so finally we've solved this problem where bridges can now participate in encrypted rooms without the overhead of having to call a sync request to get things like device messages down and it's a little bit technical but while the existing forms have had with end to bridge encryption is that device to pull down information about encryption you had to hit sync APIs on the server which was a polling action and the polling action across different users meant that it was a bit slow and all the bridge bot had to be in lots of rooms which you didn't need to be in and the new stuff is all happens down the transaction API which is a push technology which means that the hope so does sends data to the app service which is meant to be a bit better for the whole performance footprint so that's looking pretty good the botus decay itself has landed all the components inside its beta branch and I've been using it already it's actually powered by the message rusts SDK under the hood which is pretty cool in itself but yeah one of things to look out for is the fact that maybe this year most of our bridges will gain into bridge encryption by default which is just really cool the fact that we don't have to worry about whether a room's encrypted or not it's a bridge to it and of course one the downsides to end to bridge encryption is that and it's not fully encrypted but we're hoping at some point that encryption becomes property to weave these various networks if they all do not technology so we'll see otherwise that's my talk I'm hoping there'll be some questions but basically that covers everything I plan to talk about thank you
you Professor Hello and welcome to this talk, build a bridge today how we do build bridges, an we're going to see later with why bridges command interfaces are not great. It's pretty widget but we try to make sure that all the calls that maybe this year most of our bridges will