Hi, FOSDEM. My name is Woutjof Haust, and I'm a FOSDEM organizer since 2010. I've been involved in various jobs, most recently in video work. This talk is about the FOSDEM conference infrastructure, the video work, but also the way

we actually schedule the conference and organize the conference. What will we cover? The software and methods for FOSDEM, both in the online and offline variants. We may skim over some details because some things are quite involved or quite complicated, but in such cases,

we will point to past talks that will go into some more detail where possible. What we won't describe is the network infrastructure that is used for in-person conferences or the monitoring that is being set up. If you are interested in that, I

can recommend the lightning talks that were done by Richard Hartman from 2016 through 2020 in the lightning talks room at the in-person FOSDEMs. Those were very interesting talks. They are recorded. You can watch them on video at FOSDEM.org or in any of our other channels, but they're not here.

We won't cover them in this talk. What is the infrastructure that we have? It looks a bit like this in our offline event. First and foremost, there is PentaBath, which is our scheduling tool that we use to communicate

with our speakers. They submit their talks there. They get approved or rejected through PentaBath, and then these talks can be scheduled on the schedule either by us for the main talk rooms or by Devroom organizers where relevant.

The output of PentaBath goes to the FOSDEM website, which builds the XML file and the whole schedule section. That XML file is then used by other tools in our workflow, such as the scripts that create the booklet basis.

It gets its information from the XML file. The volunteer management tool, which we use for handling volunteers, some jobs are related to things that are on the schedule, and therefore it needs to know the schedule. So it pulls that from the website. And also S-review, which is our video review and transcoding

tool, also requires the XML data, which it gets from the FOSDEM website. Those are the tools that use the scheduling and other metadata, which is referred by the blue arrows in this diagram.

The red arrows represent video information, video data, which comes from the in-room infrastructure, such as cameras and screen grabbers. They just forward that to Voctomex, which is a tool written by the CCC that we'll cover in slightly more detail soon.

Voctomex then combines the streams into a single stream, which goes out to the streaming infrastructure. Streaming infrastructure then can be used by people who are not in the room to watch the live stream. It also records the live stream to disk,

and that recording is used by S-review to create the post-processed videos that go out to our video mirror network, as well as to YouTube in the past. The Voctomex tool, the mix that is created by Voctomex, can be controlled by a video control tool that we wrote.

It's a very basic tool that allows you to choose between a few presets. Use of this tool is optional. We don't require Devrooms to use it. And we're perfectly happy for Devrooms to just stay on a single, the default preset. But if they want to, they have the ability to make changes.

For an online event, it looks slightly different, but not much different. There are a few blocks that go away. The in-room infrastructure is gone, because there are no rooms. As a result, there is no need to choose

the... There's no need for a manual choice of the streaming layout anymore, so the video control tool is also gone. And finally, the booklet scripts are not really necessary anymore either, because we don't really create booklets. So those are the three things that are gone.

What's added is Matrix, the Matrix user interface that we created to allow people to participate in an online FOSDEM. It's the output of all that info, of what comes to our streaming infrastructure.

It just goes to Matrix this time, and it is through Matrix that people can watch the stream. Other than that, the tool just stays the same, except that we also use S-Review as the pre-processing tool rather than just the post-processing, because whether you're pre-processing or post-processing video,

the requirements are very similar, and so we just kept using S-Review and slightly modified it so that you could also use it as a pre-processing tool rather than just the post-processing tool. But it's mostly the same. So to go into some more detail of all the tools that we have, let's start with PentaBoff.

PentaBoff is a tool that we use for scheduling all the talks. As I said, it's done either by us ourselves for the lightning talks and the main tracks, but it can also be used by Dev Room organizers to schedule their talks. It was originally written by the CCC, the Chaos Communication Congress, but our requirements are somewhat different,

and so therefore we've made significant changes to that tool. The problem with PentaBoff is that it is getting quite old. It does the job that it needs to do, but there are some warts left and right that make it somewhat difficult. For instance, there's one page that we use to do permissions management,

and the instructions on that page are load the page, let your browser use 100% CPU time for a few minutes until everything is loaded and the JavaScript is finished. Then make the changes you want to do, click the Save button, and then wait a few minutes again for everything to be stored. This is obviously not optimal,

but it works once you know how to use it, but yeah, there are downsides. For this reason, we would like to replace PentaBoff. Unfortunately, because it is hooked up to so many things and because so many things depend on the output of PentaBoff,

replacing it is not an overnight job. This will take quite some effort, and we've actually been wanting to do this for many years, but we just haven't been able to do so just yet. What does PentaBoff look like? If you look here, this is the PentaBoff interface used by Devroom organizers and main track organizers.

I'm looking at the talks that have been accepted for this conference organization, Devroom. As an organizer, this is the interface that I see. Let's look at this talk, for instance, which is the talk that you're watching right now.

Then I have all the abilities that I want to make changes here. There's an abstract that I can see. There's ability to make some notes for myself that are not shown to the speaker and so on and so forth. I can schedule it. Things can be rated. People can leave feedback and so on.

This looks slightly different from the submission interface, which is the interface that speakers use, but in essence, the idea here is that the PentaBoff tool is used by us to schedule talks

and to handle all the talk data. From PentaBoff, everything goes to the files and websites. PentaBoff itself doesn't really talk to many other things. So the Fosn website itself is a static website. That is, all the files are static

and are compiled by Nanoc, which is a Ruby-based tool, like a static compiler to create a static website from dynamic data. The source, or at least the mirror of the source, is available at GitHub on this URL. The website repository, the Nanoc itself,

will pull all the data it needs from PentaBoff to build the computer parts, like the schedule and other things. The way we've set it up seems to work quite well. Although the output is not ideal in every location,

like the schedule itself becomes large and somewhat difficult to navigate in certain cases, and could therefore certainly use some usability improvements, but I don't think we will replace the way we build our website anytime soon. It seems to serve our purposes, at least that's how I feel about it.

Another tool we use is for volunteer management, which looks like this. Sorry, I forgot. I was going to show you the Fosn website itself. This is basically the way things work. Um, let's see.

There was, ah, yes, in the content directory, I forgot. I always forget that. There is a news thing, for instance, and there are some HTML files. They're not really just HTML files. They are slightly different. You can see there's a title here, and there's a fault there, and basically that's how this works, so it works quite well.

Um, the schedule is somewhat more involved, but anyway, our volunteer tool looks like this. So volunteers have the ability to go to a certain job, and they want to say, oh, I want to do, I want to work on this job,

which is hosting for a talk. Let's go to this one. You can see where volunteers have already been added and where they are still needed, so that the volunteers have signed up for this talk, but not for that talk yet, so we would need volunteers there still. So the idea here is that volunteers can say,

can sign up to a job that they want to perform without one of us having to coordinate that, and the tool will let volunteers know if somebody's already there or not, and will show where there is, where work is still needed. It is a Django-based tool. It imports a schedule from the Fosden website

to create the jobs that are necessary per talk, and then volunteers can sign up for whichever job they want to do. This tool works fairly well, but then the requirements are not very complicated, so it's just a simple tool.

The video infrastructure, our central tool in the video infrastructure is Voxomix. Voxomix is a tool written by the CCC as well that allows various different video streams to be combined into a single stream. It is a software broadcasting infrastructure.

Voxomix runs on laptops in the network operation center at an in-person Fosden, rather than in the room as in some other conferences. We do not use the GUI tool for Voxomix called VoktorGUI, but instead we use that web interface,

the video control tool that I spoke of earlier. The default layout that we create is a pick and pick setting, but if people want, they can change it to a different layout. The reason we don't use VoktorGUI is because VoktorGUI itself, while it's easy to use,

does still need some training, a short amount of training, but we do not have the ability to train our dev room organizers at all, and therefore we try not to do that. We try to keep it as simple as possible. The in-room infrastructure is basically a wooden box

that was laser-cut that we created ourselves, and then some hardware inside is mostly an arm SoC to combine everything, and then there's an HDMI to SDI converter, so because the encoders we use require SDI, and that encoder is made by Blackmagic Design.

It's an H.264 encoder. For the cameras and other AV equipment, we rent most of this because buying that is just too expensive. As I said, we try to keep everything as simple as possible because training more than 20 teams is just not possible. To give you an idea of what these boxes look like,

these are the first year when we used the wooden boxes in 2017, when we were doing a final Ansible run over a small network here. It's a bit messy, but you can see all the rooms have two boxes, one for the camera and one for the speaker capture,

like the slide capture device. If we open one, this is a Fosden box that's on the inside. Right here, you can see the H.264 encoder. This is the SoC, RMSoC, I believe it's a banana pie.

That is the HDMI to SDI converter, and then here there's a network switch as well because you only want to output one cable and not three.

Once everything has been recorded, the video goes into something that I wrote, S-Review, which is a Perl-based web app, and then a set of scripts. S-Review sees the video as a timeline and then uses the schedule and a lot of crowdsourcing to extract the content that we actually care about.

I did a talk about S-Review at Fosden 2019 and that recording is available. Which you might want to look at if you are interested in using S-Review. For Fosden 2021, we added upload functionality, which was written in a way that it would allow people

who want to do manual editing of their talk to upload their talk without requiring manual intervention by the administrator. And that is slightly abused to allow uploading and processing of pre-recordings.

So like I said, the manual overrides is something that we have had requests for in previous Fosdems, but yeah, it's not necessarily something that we absolutely need for an online conference, obviously. S-Review looks like this.

If you are a speaker at Fosden, you would have seen this already. And this is the upload form where I can select a file. This is the upload form for this talk. So this is before this talk, obviously. Basically, you select the file and then you click upload. It's very basic infrastructure. It doesn't use all the HTML5 spiffiness

that allow you to give a progress bar, et cetera, but it does the job and that's what's necessary. There is an overview page that shows you all the talks that the system knows about and what their state is, waiting for files or files that still need to be uploaded.

So there's quite a lot of them. Preview are the talks that have been uploaded, but that require a review and then uploading speaks for itself. And then there is DOM. I mean, there are many states in the system. There are some that we don't see here, but basically the idea of SREview

is that all the talks go through the system and then eventually something rolls out. It is written in such a way that as much of it will work automatically as possible. What have we changed for an online Fosden? Well, like I said, the prerecorded videos,

they are uploaded into SREview. People say they go through PentaBaff, but it's actually not true. What we do is we distribute the URLs for uploading a video through PentaBaff. So speakers have to click on that URL and then they get an SREview page

where they can actually upload. From SREview, they are then synced to the Voxel Mix machines who are on a cloud service. In CASU, this is actually Hetzner. Voxel Mix is no longer controlled with the video control tool, but instead it is controlled by a bot that knows the schedule,

that knows the video length and will automatically switch the input from the pre-recording to the JITC room, et cetera, et cetera. The user interface is based on matrix for people to look at a talk and JITC for those people who want to be part of a talk

or want to join the live stream. If you're interested in more details here, then you should look at last year's matrix talk where the elements people will give some detail in the explanation on how all that works together.

What if you want to do this yourself? Well, everything that we do is in Ansible except unfortunately for the matrix bits, those are separate. If you want to get started, then this repository and the infrastructure repository is where you would start. It contains everything, not just the online bits,

also the in-person infra and also things like the network configuration and the monitoring, well, not the configuration obviously, but the way that the network is set up and then the monitoring is also in there. The video team hangs out at Fosden Video,

obviously not the network team, they are in a different channel, I should have added that, my mistake. The documentation is not entirely up to date everywhere, I think people know how that works, hence this talk, but if you have any questions and you want to use some of the infrastructure

that we use, then you definitely can contact us and we'll help you out with pleasure. Like I said, the network team is not in Fosden Video there, but you can also reach the team at just hash Fosden at Libra chat. And that's the end of my talk. Thank you for listening.

If you have any questions, I will be happy to answer them, thank you.

All right, that is the end of the talk. I'm not sure if there's anything else at this point, there were a few questions. So one question was whether these wooden boxes are used for the remote event, the answer is no, we don't use them for that

because I mean, they're the in-room infrastructure, we use them to hook up cameras and frame grabbers for slides and that kind of thing, which you obviously don't need for an online event, so no, we don't use them there. But yeah, the talk wanted to show both online and offline workflow, so that's why I showed that as well.

Another question was, I assume that alternatives for submissions and evaluation have been explored such as pre-talks. Yes, we have been looking at things, but of course there's a difference between looking at something and actually implementing it. We've done the first, we haven't done the last yet.

It's just a matter of time. There are also a few missing features in the options that we explored. I think at some point we also explored FRAP and maybe a few other options too, but every time we came up with some requirements that we had that weren't implemented.

So yeah, hopefully we'll get there at some point. We still have some time. If people have questions, more questions? Yes, I see a question about do people

use their webcams and microphones or do we have some kind of minimum standard recommendation? For cameras, we don't really tell people what kind of camera to use. Usually people just use webcams and that kind of thing. We do have like a standard of encoding,

like try to get it at 720p and try to do about two megabits and we are working with MP4, so if you do that, then we don't need a transcode, that kind of thing. But we don't really have any requirements for the webcam. For audio, we do recommend that people try to use a headset like this one

rather than just a microphone in the laptop because that creates a lot higher quality than if you try to do something in a laptop. And another thing that we try to tell people not to do is use a Bluetooth headset because Bluetooth can sometimes go back to eight kilohertz

rather than higher bit rates in audio and that is actually quite audible. It's not very nice to listen to a talk that way. So those recommendations we have, but yeah, that's it. Other questions, lots of questions.

Two years ago, there was a project going on to replace the capture cards by an own designed one. Did that work in the end? I'm not aware of that. I don't think it happened. I don't think it happened in detail. I do know that DEPCONF was looking at that and they did go with a few options there,

but I think FOSDEM was not ready to jump on that chip yet because I mean, there's a difference. I mean, DEPCONF, there's three rooms, FOSDEM, there's 24. So there is a difference and for that reason, I think we didn't go with the designed ones because I mean, what we have works and if it didn't broken, don't fix it, I think is the situation there.

Could you use big blue button instead of JITC? I'm sure you could, but we just ended up going with JITC partially because I mean, I'm a member of the DEPCONF team as well and we gave that a try and that seemed to work. And then when we talked to the people from Matrix, they said, well, we already use JITC in Matrix 2

and we integrate it there. I don't know if it's possible really to have big blue button integrated in Matrix like JITC is, but I have, I mean, if you really want to know that I think it's probably best to talk to the elements people, they might know better. I think I've had all the questions

or am I missing one? No, I think that's it. Right, I can't come up with anything else.

Kyle was going to join me for a final goodbye to everyone. Don't know if he's here yet.

Well, I can say, oh, there you are. Hi, Kyle. Sorry? It took a while to join for some reason. Oh, okay, yeah.

Technical issues always great, isn't it? I definitely can say that I personally think that this first time of a conference like this, like a conference, organizing conference was quite a success. And I think we had some interesting talks and we should probably try and get more of this in the future.

So thank you for helping me out with this. I don't know if you want to say something. Yeah, definitely. Thanks a lot for kind of having the idea running with it. It's definitely something I'd like to see happen again and maybe grow to a bigger dev room, maybe have a full day's worth of talks next year.

That'd be great. Yeah, yeah. If we get people who are interested in submitting talks, of course, because otherwise. Yeah, I'm sure not everyone wants to listen to us prattle on for eight hours. That is true, yeah.

All right, I think we're ready then. Unless there's something you want to say or somebody else want to say. Nope, I think that's it for me. Thank you everyone for coming and joining and making this dev room work.

It's a lot more interesting when there's people that actually attend. We did just get another question. Could some peer-to-peer help for the streaming or would it only make things more complicated? I think it would make things more complicated. Streaming is something that requires large amount of bandwidth

and you want it at low latency and otherwise it would be an issue. I think it might be problematic to try and go peer-to-peer at that. But yeah, I don't know. I would say go ahead and try and if you make it work, more power to you.

And with that, I think I'll join you in thanking everyone and calling it a day. And then hopefully next time. Bye. Yeah, cheers.