Hello and welcome to this talk, another matrix talk about the next generation of matrix interfaces.

I'm Will, or Halfshot, you might see me around, I tend to work on the bridgey side of things, and today I'm going to be working on a new thing, interfaces, relating to bridges and integrations. So this presentation is going to cover how bridge interfaces work today, and how integrations work in such a way that users can interact with them,

so bot commands or other sort of forms of clicky button type stuff, how we're going to try and change things. Preface, this talk is very much about experimenting and trying something new. This isn't an officially sanctioned matrix thing we're going to do tomorrow, but this is certainly something that I've been experimenting with the last few weeks, and I'm hoping this will change the way we do integrations going forwards.

Now finally, a little bit on what's new in Bridgeland, just a little bit of news to catch up on that really. So first off, let's talk about bridges. I won't go into too much detail, because I'm pretty sure most of the audience have a vague idea of bridges, or overly building bridges, usually one or two caps.

For those who don't know, bridges basically are a gluing mechanism for connecting remote services like Telegram or RFC into a matrix. Typically they tend to exist next to a home server, a bit like this. So you see here we've got our clients, we've got our home servers, and we've got application services. The application service is where the bridge usually lives.

It's connected usually to at least one home server, so it sort of works in a meshy style system. You see here, this is our example network of a bunch of different siloed services. You've got your Telegram, RFC, Slack, and Gitter, and so on. And they all sort of exist separate from each other. Some of them might connect each other, but there isn't a general meshing system.

And of course, this is what matrix provides. Everyone's fairly aware that at this point, matrix is the glue between them. I mean, if you've come to Fostom through IRC, then you already know this, because you're talking through IRC to matrix, or Slack perhaps. But yeah, generally speaking, this is our bread and butter of how we do stuff.

So this is bridges across section. An example of how we build a bridge today, how we do build bridges. An example of a Twilio bridge. This is actually one I used in my previous talk at ComCom, where I built a bridge in about 100 lines of code to prove that you could basically build an SMS bridge with very little experience and, you know, just an hour of your time, really.

In this case, I'm just showing off that we've got the client on the far end, which is TensorFlow users used to interface with the home server and the bridge and so on. And then the bridge sits in the middle there, talking between the home server and the remote API. What's interesting here, basically, is that the user never directly interfaces with a bridge. There's always a home server in the way.

And so to contextualize, this is some of the struggles that we're going to see later with why bridges sometimes are very difficult to control. And now we're going to talk a bit about interfaces. So interfaces in what we're using here is to mean how do users interact with services that aren't other humans, I guess to put it plainly. If you have a bot and a chat or a bridge or some other service where the user needs to send a command or do a thing, how do they do it today?

And how they do it today looks a bit like this. We have commands, basically. I've probably built up the majority of our communication team, these services, is all through command line interfaces or whatever you prefer to call them.

They've been with us for a very long time. I think probably since the early days of matrix until mostly up to this day, we still use commands to control bridges. There's a few cases where we might have special metaphors like you can join. You don't have to send a command to join a ISC network. You can use aliases to join them. But by and large, you would do anything particularly complicated like set your NIC serve password

or if you want to do something like join a unbridged room or something particularly clever. That all has to be done through commands, which kind of sucks because command interfaces, they're not great. It's pretty painful, honestly, to use these things. To go into a bit more, you can see that they're pretty intuitive for new users who have never used a command interface before.

Most people today that are joining Element are probably not used to using sort of TUI based stuff, especially if they're coming through one of the mobile apps or, you know, looking through Facebook or something. They tend to be doing using stuff through clicky button stuff or invites or QR codes.

And that's perfectly fine. We want to support these users as much as we want to support the more hardcore users out there. The problem we have is as soon as you want to escape the bubble of the simple matrix communication chat or the room directory jumping into a room, so you can get more complicated than that and you actually want to start using the more powerful features,

you're suddenly hit with this wall of commands. And I don't think it's what people want to see. Matrix clients are built by graphical interfaces. You know, a lot of our clients are quite clever and they look nice and it's great. But then suddenly it's the issue of using integration. So it drops down to the floor of now you're just typing commands, typing in help commands, looking through wiki pages.

And that stuff just, it ruins the experience, I think, to some degree. We shouldn't really expect matrix users to be typing bot commands. This isn't to say that we should kill them off tomorrow. This is to say this should not be the default. This shouldn't be the way we're expecting users to use our stuff. So how do we fix this? We could define a matrix spec to cover the things that bot commands cover today.

So we could add extra commands to do things like rolling a dice, or we could do, we could try and cover every possible use case for the very more extreme uses out there. But typically this doesn't tend to work because it increases the bloat with the spec. So every little feature you want to add to maybe do your own small little game on the side,

which really means the client has to go and implement the feature in some form on their stuff. This might be the best for open standards and ensuring that, you know, everybody's on the same hidden sheet potentially, but it has the effects that most clients probably end up implementing it.

You'll have a sort of split brain between maybe element implementing this feature and then a bunch of clients not implementing it. So it doesn't necessarily solve the problem for the smaller use cases. Certainly for the bigger things out there, we should try to ensure respect. But for the smaller little extra features, it might not be the suitable way to do it. We should try and, we could try and define a common language to define all the extra functionality.

So some sort of right at no XML based language where you could define your forms and things and buttons and some sort of scripting language where you could just interpret the command and then send it back. But we have that already. This is basically widgets, matrix widgets. So people don't know what widgets actually are because they actually only end up in, so far not in, most clients don't implement them.

But this is hopefully going to change at some point. But widgets in a nutshell are HTML interfaces or HTML pages which you can insert into a matrix room. They're completely sandboxed on the matrix of clients, so they have an API that can talk through to the client, but they don't fundamentally have access to all the databases and bits and bobs of the main client.

They're a separate entity. And the idea is basically you can build your interfaces or hook in your existing website or something like that into your interface and just have it sit alongside your matrix room. So if you had a collaborative document you wanted on the side of your chat, you can input that as a widget and redo that sometimes, a YouTube video or something, or an interface to control your matrix bot or something.

The other nice thing is it has a capacity negotiation system. So if you want the ability to limit what the system can do, so it can't read your events or it can't read your account data, by default widgets have a very limited view to your account.

But if you want to grant fine-grained control to your account, that is a thing they support. And actually, it's a super powerful feature when you consider that you don't necessarily want to be able to have full access to your account. Architecturally, they look a bit like this. You have your client, again, a bit similar to our last diagram, but you have a widget that sits sort of on top with its own widget API.

And in our example, we'll see a bit later, there's a REST API, which you can have a REST API in your widget and the bridge. The widget also has access to some of the data that you'd have on your matrix account, which it grants through the widget API. So you can sort of see here, it's not a direct connection to the home server. The widget can't directly start calling client server API stuff, at least not out of the box.

You could potentially provide a access token to your widget, but we try to make sure that all the calls to the home server go through the client to avoid sort of any potential dangerous actions. But yeah, typically this is how we would build a thing. So the next one I want to talk about is how do we invite users to a DM over a bridge?

This is a common problem we've got with bridges at the moment, is bridges can't yet provide a mechanism to invite users who have never spoken on matrix. We're perfectly able to invite users who are already chatting on matrix, so I can always invite someone from my RSC conversation to a chat at DM, that's fine. But if the user has never used the bridge before, how does matrix show a UI for users which had never set up a profile, never interacted?

You'd have to have some sort of directory system. This can be achieved with other mechanisms, but one way it's really easy to achieve is simply a widget in your room, somewhere where you can just type in a name, the bridge will search for them and then return the result.

This is actually what we're going to demo just now. So if I switch over to my lovely Slack demo here, you can see here that I've got a matrix set up, an element. This has got a few things on it, hookshot bridge, dragon bridge and a Slack bridge. You can see here we've already got a bunch of things in here. Crucially, we have this bridge control widget.

I just want to quickly show off the widget itself. Not the actual widget, not the widget's UI, but actually the content of the widget. You can see here we've got a, if I zoom in a bit so it's actually visible, you've got a UK Harshaw Bridge Room subscriptions thing, which basically says, hello, this widget provides a feature for your bridge.

In this case, it provides an invite widget feature. And now I'm going to see how that actually looks. So I go to start chat and I reply and look for people. You see you've got the usual stuff, but interestingly, you've got this bridges section with a hookshot settings room, which corresponds to this guy here, but also a Slack bridge room, which corresponds to our Slack bridge. So if I click through to this, and I actually run out of permissions to,

again, this is our permission system granting permissions to verify that my user ID. We can hit here and then sign in presented with this lovely dialog box. I will admit my design credentials are not as good as my bridging and stuff, but generally speaking, this is a simple UI on how to search users on Slack.

So what this is actually doing is rendering, Element itself is rendering a very generic window, which the bridge then fills in its own UI. So this bit here is all running on top of the bridge and this outer bit is Element. So if I then search for FOSDEM, you can see here there's this user here called Will FOSDEM.

Never spoken to him before in my account, but now if I click through now and click this guy and then type here, and then go to here, I've now got invited, I've now started dealing with this person. And then if I then send a message saying hello, and hopefully if this person is actually going to respond to me, we should find out. Here we are. I've said hello to him over Slack, and if I then say hello back,

just to prove the fact that this is definitely a live demo, you can see here that I've sent a message back. So proof here is you basically can then look for people through this lovely system and it also works, I mean, I can also prove that it finds me and myself. So that's cool, if I did Will, does this, because in our particular demo for the Slack bridge,

it actually searches both your name and your display and inserts so it can find people through multiple directory keys, if you will. So that's pretty cool. So that's our demo for how to do widgets, and a very basic demo of how widgets actually work in a bridge context. Of course, you could do way more than that.

We could have bridge settings rooms and things like that, or we could have images appearing in the rooms. One of the things I've always loved to have the idea of is a side-winding widget on one side of a room on the other side, so you could see news or some sort of extra data

that your room lesson would be able to provide on its own. But this is the idea of where you could extend this further if you wanted to, or add multiple sort of invite types. So let's talk about the fallbacks, which you've seen me leak slide just now. So one of the things that's still missing is obviously that not all clients support widgets.

So we're trying to think about how we can do fallbacks for this. One idea is the idea that since the widget is just calling REST APIs, we just expose those REST APIs and the authentication mechanism, and so the client itself could call these. And this would be good for the case where you are a client that doesn't really want to bundle a whole web rendering engine but would love to be able to support, I don't know, the IRC bridges, the widget API.

And so what you could do is obviously inspect this open API spec and then potentially build out an interface on top of that. And that's one idea we're exploring, some sort of limited form of also form building. We want to be very careful there and make sure that you still have the ability to do nuanced things and more than just a common basic form. We don't really want to start speccing our whole forms in matrix thing

given that's a solved problem elsewhere. And of course the one thing we want to ensure that we still have is bot commands as a fallback. And it's in a way that in Linux you'll have, you know, fallback commands, so pretty much everything you do in the UI or in the GUI, the same reason we want to make sure the matrix still has that fallback for people who don't want to present the UI,

which is still a very useful thing to have. By the way, yeah, we're hoping this will be a way in which we can sort of build much richer integrations into existing clients. So next steps, how do we take this further? So we've got some MSCs on the way, as you would expect with most of these things.

There are two MSCs currently out there and how we can expose user IDs from widgets to clients. So you can start an invite or start a DM. I've got one of them, 3.6.6.2, which is very particular about user IDs in the texture of how it conveys intents to the client about what you plan to do with them. There's already an existing MSC, 2.9.3.1,

which handles room IDs and events too, so it might be one of those will probably end up winning and the feature will end up in one of these ways. There's also the widget spec itself, which is in the draft also massive, huge. Congrats to Travis for writing a massive, massive document about how you do widgets

and it's like pretty complete, but just not landed in the spec yet. For what it's worth, most of the widget stuff is fairly stable at this point in terms of actual usage. A lot of widgets are already in use by Element in various places, so we're pretty confident of the actual technology itself. We also need to land the bridge components for this demo.

With what I demoed there was a bunch of stuff that's in the branches. Most of it's in a draft state. The next thing I'd like to do is take this further and have it land as part of the core, make sure it's a bridge SDK, so we can start using it. I'd also like to land the web stuff there. As you saw there again, most of it was branches, but perfectly reasonable.

One of the things I'd like to do with the web implementation in particular is have more settings view so you can see which bridges you subscribe to and also how to control which bits in the UI they show up in. We'd like to develop the fallback solution we talked about earlier, so it's old clients don't get left behind.

A crucial part of this whole thing is that we don't lose clients, because only one thing worse than trying to build a better UI is you end up killing off a whole bunch of your existing user base by ending up having no one who can use your stuff. So the only way we'd ever progress with this widget stuff is if we kept access to all the existing commands and made it still as easy to use as possible.

So that'll be one of the challenges we'd like to look at first. And of course, if all of this stuff happens, we'd like to make sure that the IRC bridge gains it and then the Slack bridge obviously improves its support and so on and so forth. We'd like to make sure that eventually all the bridges become as ergonomic, because this can be. So yeah. Finally, the final thing we'll talk about is basically what's been happening in Bridgeland already.

So there's a few things out there, but I think we'd call that N2Bridge encryption is now formally landed inside Synapse and the Bot SDK. The Synapse stuff is very, very close. There's a few merges to happen before it's in the stable, but I've played with it and it's absolutely brilliant. And so finally we've solved this problem

where bridges can now participate in encrypted rooms without the overhead of having to call a sync request to get things like device messages down. It's a little bit technical, but while the existing problems we've had with N2Bridge encryption is that to pull out information about encryption, you had to hit sync APIs on the server, which was a polling action,

and the polling action across different users meant that it was a bit slow and all the bridge bot had to be in lots of rooms, which you didn't need to be in. The new stuff is all happens down the transaction API, which is a push technology, which means that the Hope server just sends data to the app service, which is meant to be a bit better for the whole performance footprint.

So that's looking pretty good. The bot SDK itself has landed all the components inside its beta branch, and I've been using it already. It's actually powered by the Metis Rust SDK under the hood, which is pretty cool in itself. But yeah, one of the things to look out for is the fact that maybe this year, most of our bridges will gain N2Bridge encryption by default, which is just really cool. The fact that we don't have to worry about

whether a room's encrypted or not, it's a bridge to it. Of course, one of the downsides to N2Bridge encryption is that it's not fully encrypted, but we're hoping at some point that encryption becomes proper E2E with these various networks, if they all adopt R10-IG. So we'll see. Otherwise, that's my talk.

I'm hoping there'll be some questions, but basically that covers everything I plan to talk about. Thank you.