Running a Hybrid Event with Open Source
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 542 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/61623 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2023290 / 542
2
5
10
14
15
16
22
24
27
29
31
36
43
48
56
63
74
78
83
87
89
95
96
99
104
106
107
117
119
121
122
125
126
128
130
132
134
135
136
141
143
146
148
152
155
157
159
161
165
166
168
170
173
176
180
181
185
191
194
196
197
198
199
206
207
209
210
211
212
216
219
220
227
228
229
231
232
233
236
250
252
256
258
260
263
264
267
271
273
275
276
278
282
286
292
293
298
299
300
302
312
316
321
322
324
339
341
342
343
344
351
352
354
355
356
357
359
369
370
372
373
376
378
379
380
382
383
387
390
394
395
401
405
406
410
411
413
415
416
421
426
430
437
438
440
441
443
444
445
446
448
449
450
451
458
464
468
472
475
476
479
481
493
494
498
499
502
509
513
516
517
520
522
524
525
531
534
535
537
538
541
00:00
TwitterOpen sourceEmailMatrix (mathematics)Event horizonQuicksortOpen sourceSinc functionPosition operatorPresentation of a groupInteractive televisionKernel (computing)Computer animation
00:47
Computer networkPiQuicksortKernel (computing)TrailData conversionMultiplication signFocus (optics)View (database)VideoconferencingPresentation of a groupBitInteractive televisionException handlingStructural loadEvent horizonMatrix (mathematics)Computer animation
05:02
Inclusion mapElectronic meeting systemVideoconferencingService (economics)INTEGRALKernel (computing)Structural loadTouchscreenSoftware developerWellenwiderstand <Strömungsmechanik>Open sourceDebuggerOnline chatSoftware maintenancePresentation of a groupRevision controlMultiplicationDesign by contractMatrix (mathematics)Interactive televisionTelecommunicationWeb 2.0Computer configurationSurjective functionSet (mathematics)Real-time operating systemCodeFront and back endsTrailPhysical systemReal numberProcess (computing)Streaming mediaGreen's functionWeb serviceComputer animation
11:08
DisintegrationPasswordAuthenticationFigurate numberEmailMultiplication signQuicksortSpreadsheetServer (computing)Self-organizationAddress spaceINTEGRALNumberWebsitePhysical systemKernel (computing)Front and back endsComputer animation
12:52
Standard deviationMassZoom lensSystem callMultiplication signGoodness of fitInteractive televisionEnterprise architecturePixelVideoconferencingLaptopTrailString (computer science)WebcamINTEGRALComplete metric spaceView (database)TouchscreenLoginBitModule (mathematics)Online chatYouTubePhysical systemData conversionVisualization (computer graphics)Streaming mediaData compressionProjective planeOpen sourceBand matrixQuicksortStructural loadLipschitz-StetigkeitCore dumpNumberFigurate numberEndliche ModelltheorieTape drivePoint (geometry)Computer animation
18:00
Matrix (mathematics)Embedded systemServer (computing)Online chatWhiteboardDecision theoryINTEGRALReal-time operating systemMultitier architectureBlogScaling (geometry)Default (computer science)Software testingNumberFront and back endsEmailMatrix (mathematics)QuicksortProxy serverSequenceHybrid computerData compressionTrailRemote procedure callComplete metric spaceSign (mathematics)PressureModal logicFigurate numberGUI widgetOrder (biology)Form (programming)Structural loadSoftware developerKernel (computing)Right angleWeb 2.0Thread (computing)Address spacePoint (geometry)Reduction of orderClient (computing)Tape driveNeuroinformatikPhysical systemBefehlsprozessorType theoryTelecommunicationFlow separation2 (number)Single-precision floating-point formatCartesian coordinate systemComputer animation
26:10
Embedded systemOnline chatMatrix (mathematics)Beta functionWebsiteFrame problemRemote procedure callKernel (computing)PlanningLattice (order)Personal area networkRevision controlQuicksortMultiplication signWave packetVideoconferencingOperator (mathematics)Latent heatView (database)BitStreaming mediaFront and back endsSystem administratorInternetworkingFunctional (mathematics)Different (Kate Ryan album)MultiplicationSet (mathematics)Electronic mailing listStrategy gameFilter <Stochastik>Data conversionProcess (computing)Video projectorVolume (thermodynamics)Matrix (mathematics)ImplementationOverlay-NetzPatch (Unix)Selectivity (electronic)Online chatFilm editingPresentation of a groupoutputUltraviolet photoelectron spectroscopyGame controllerComputer animation
34:20
Gamma functionVideo projectorWeb browserOpen sourceThread (computing)Price indexKernel (computing)Computer iconConnected spaceMatrix (mathematics)Medical imagingGraphical user interfaceComputing platformInterface (computing)Front and back endsTDMAResultantComplete metric spaceMultiplication signInteractive televisionData conversionMilitary baseVideo game consoleStreaming mediaFunctional (mathematics)BitPresentation of a groupVideoconferencingLaptopPhysical systemComputer animation
39:37
Execution unitSimultaneous localization and mappingLocal ringFilm editingGame controllerQuicksortDependent and independent variablesINTEGRALLattice (order)BitParadoxConnected spaceWeb browserTrailStructural loadSoftware developerVideo game consoleComputing platformSurfaceKernel (computing)Web 2.0Open sourceWindowLink (knot theory)Hybrid computerWeb-DesignerComputer programmingMultiplication signScripting languageNumberPhysical systemInteractive televisionModulo (jargon)Operator (mathematics)Wave packetPoint (geometry)Uniform resource locatorMaxima and minimaOrder (biology)Presentation of a groupKey (cryptography)Computer animation
44:51
Order (biology)Axiom of choiceDesign by contractRemote procedure callCASE <Informatik>Adaptive behaviorLattice (order)Universe (mathematics)QuicksortPoint (geometry)Uniform resource locatorConnectivity (graph theory)Staff (military)Shared memoryVisualization (computer graphics)Group actionDigital electronicsData conversionPersonal area networkMultiplication signInteractive televisionOnline chatHeat transferStudent's t-testComputer animation
50:41
Program flowchart
Transcript: English(auto-generated)
00:06
So, it's been a while since I've been in FOSDEM, and one of the things that I, well, I usually talk about technical topics, but this is really me talking about sort of using open source instead of producing it, which is quite an unusual position for me to be
00:23
in. I work most often in things like the Linux kernel, I also work in open source politics, so I did a lot of kernel interaction with the Linux Foundation, and one of those pieces of interaction was actually being a founding member of Linux Plumbers Conference, which
00:43
is the conference we'll actually be talking about for this presentation. Plumbers was basically founded in 2008 as an in-person conference, and it's always been an in-person conference up to the pandemic years. It's actually organized somewhat similarly to FOSDEM in that it has tracks and what are
01:04
called micro-conferences. The micro-conference is similar to Dev Rooms, except they're really a bit more interactive, so they're really expected to do participant conversations, and usually you get a cluster of people at the front talking to each other.
01:20
The tracks themselves, when we have them, mostly tend to be presentations, so very like this, but the micro-conferences tend to be very discussion-based, so if you think about trying to do that sort of hybrid and online, you realize you need some sort of highly interactive video system, which is what we went through. One of the good things we had for Plumbers is that the conference actually produces
01:44
quite a lot of money. We have quite a lot of sponsorship, so it's quite easy to pay people to do things in the conference. One of the things we paid for in 2018 was to actually make the conference live-streaming for all of those who couldn't attend, and also to try and put all of the videos
02:03
of every presentation and every discussion we had up on the website. Obviously, to do that, we needed to pay somebody to run AV in each of our rooms, and Plumbers tends to run as about somewhere between four and eight simultaneous tracks, so we needed about 16 cameras to do this, and about between four and eight people to
02:25
stay in the room to also do it, and obviously, this then included provision of cameras in the rooms, and the way we set up cameras at Plumbers is slightly different from what you have here, because in your setup, the camera is mostly looking down at me, because
02:42
you think I should be the focus of attention, which is true, because I'm giving the talk, but when you're actually trying to record an ongoing live discussion that mostly happens between people in the audience, you also need the people who are live-streaming to be able to see it, so one of the other fortuitous things is Plumbers is always set
03:01
up with two cameras, one at the back looking down on the speaker, and then one at the front looking out at the audience, and the AV guys are actually responsible for switching between the front view and the back view, so that when we start to get conversations within the audience, the camera can actually catch them, and you can watch it, and this
03:20
actually turned out to be really fortuitous as well. Obviously, when you try and record your conference, the first thing you realize is that in a highly interactive conference, most people don't use the microphone, and it's really annoying for everybody on the streaming when you get a load of silence in the room, so persuading people to use
03:40
microphones is one of the key things that you have to do when you start recording your conference, and this is still nothing to do with trying to get it live or hybrid, this is just recording the conference. So you have to persuade all participants in the conversation to actually use a microphone, and however much money your conference makes, you will
04:02
only have a few microphones, like this one, we have two microphones at the front, I have one Lavalier microphone for a room that could possibly seat over a thousand people. Trying to run around with microphones when you're doing an interactive conversation is really difficult, as I assume the guys with orange t-shirts will be able to tell you over a beer after the event. The solution we took was borrowed from another conference
04:25
in Paris called Kernel Recipes, which are these things called catch boxes, and effectively they're square plastic polystyrene blobs that have a microphone inside that you can throw from one end of a conference room to another, and this gives you a way of actually
04:41
getting the microphone between the participants very fast, and it's definitely fun to use, and they do encourage participation. So far we've had no injuries, I should actually qualify, we've actually had no serious injuries so far with people throwing microphones at each other, but I assume there's always a first time.
05:04
And the idea was really to try and make the conference that we were running available in real time to people who can't participate, because, you know, it's a kernel development conference, we get people who want to come from all over the globe, and we always have corners of the globe, especially when we run the conference in America, that even
05:22
if they have the best will in the world and they can actually scrape up the money to come, they're often denied visas by the US authority, so it can be a really difficult problem. And for our first attempts at doing this, the communication was really only outbound, so the remote participants could watch what
05:41
was going on in the room, but they couldn't really participate. And obviously this all changed in 2000 when we got to the pandemic years. Like all other conferences, Plumbers moved to being fully online. You know, we didn't, we were actually in the year 2000, we already booked a venue up in Halifax, Nova Scotia, we had a huge
06:04
amount of fun trying to get out of that venue contract when it emerged that we couldn't actually hold the conference up there, and then we all had to scramble to try and actually do this fully online. So this is where the story becomes quite interesting, because there are 10 people on Plumbers Committee, Mike Rappaport over here is one of them,
06:22
because of the origin, we are primarily kernel developers, and because we're moving fully online, effectively kernel developers are becoming responsible for web programming, because what we're really doing is setting up a distributed set of web services. And if you know kernel developers, we mostly tend to theoretically look down on web developers,
06:46
because obviously the kernel development is at the hard core of everything. So this is somewhat a story of how a bunch of kernel developers tried to do web and failed a bit, screwed up quite a lot, and eventually got it working by the seat of their pants.
07:05
We evaluated a load of options for actually going fully online, as every conference did, they scrambled to, we discussed it a lot with other conferences, people had different ideas. We were actually under the umbrella of the Linux Foundation, and they're a much more commercial entity
07:22
than we are, we're still volunteer run, even pulling a lot of money. Their idea was just to pay somebody to do the solutions, and frankly in the early days that looked like a good idea to us, if you can just pay money to get rid of the problem, why wouldn't you? And we have enough money, we've got, you know, almost a million dollars in the bank now for Plumbers,
07:44
we could afford to spend a lot of that money to actually pay for somebody to sort out these problems, and the Linux Foundation tried to do it for several conferences, and we looked at their solution and went, you know, this is terrible, I mean, why is it so difficult and so bad trying to run a conference? And so the only way we could actually foresee getting this working
08:06
is if we just abandoned all of the commercial solutions and did it ourselves, which is much more difficult than we imagined to actually do. But the goal of this was that we definitely needed in our online conference to have two way video interaction in real time between
08:25
the participants, that was the goal, and there was no real conference system that did that. If you've tried holding conferences over Zoom, you can just about do it, but things like BBB actually do an awful lot better than this. And the other thing we discovered is that when you're running a collection of web services, it really is all
08:45
about the integrations. Open source gives you a lot of excellent pieces, so we use big blue button, we use Indico for the conference scheduling, eventually we use Matrix for the track, we didn't use it in the first year, but there's also a downside
09:01
to that. Whenever you try and integrate web services, you always write a set of glue around them that needs to be maintained, and if there are a load of disparate services that aren't used to talking to each other, it does the impedance matching between the services, and as the years go by, and remember this is a conference, so we put it on once a year, every year you upgrade to the new versions and suddenly your glue
09:24
doesn't work again. So it always becomes a maintenance nightmare to get the glue working. So, if you ever came to the Plumbers' Conference, the web front-end that you'd first see, if you go to meet.lpc.events, is a piece of code that John Corbett wrote in Python
09:44
and Django called the Linux Plumbers' Conf front-end, LPCFE. Oh, and I should add that these things here are clickable URLs, so if you go to the presentation and you want to know where it is, you can just hover over it and it will tell you.
10:02
Obviously green light is actually the way you log people onto BBB, so by replacing green light and big blue button, we've effectively taken on the job ourselves of doing integration, authentication, user management, and pretty much everything else. This is
10:21
actually a major replacement of services on big blue button, and the reason we did this is because we needed an integration with our conference timetable. We just wanted the conference timetable to display on the first screen with a little button that said click here to go to this session. It's very like you have in FOSDEM with the matrix
10:41
chat. If you just click on the integration of the matrix chat, it brings up the live stream and you can see what's going on in the conference. We wanted it to work pretty much as it does here, and green light just wouldn't do that for us because green light was designed to be a teaching front end for a single classroom. It wasn't really thinking about multiple tracks, multiple track rooms, and timetable integration.
11:10
By getting rid of all of the attendee credentials, we actually had to integrate with some sort of authentication server that could be distributed. Being kernel developers, we chose LDAT because
11:24
it's the oldest one and it's the one everybody has the most trouble with, so we should be able to use it, right? The credentials were the email address that you used to register with the conference, which is easy, and the password was just the confirmation number because fortunately the Linux Foundation CVAN system spits out these long confirmation
11:44
numbers and they're all guaranteed to be unique. So that's obviously really easy. Problem, again, integration. The CVAN system will not speak to anything outside CVAN, so if you register for the conference and you don't automatically show up in the LDAT database,
12:04
what has to happen is I have to go to the CVAN website and export a spreadsheet of all of the users, figure out which are new ones, and then feed the details into the LDAT server, which, I mean, as an offline activity, it sounds really simple. It becomes really complicated when you keep getting people who don't register until the last
12:24
minute of the conference, turn up halfway through a session, want to register, and then immediately be able to see what the hell's going on, and they have to wait for I don't know how long for somebody to actually put their credentials in the database, and when you're running a conference, pretty much all of your attendees are running around
12:42
or all of your organizers are running around full-time. Nobody has time to tend the back end of the database, and that became a bottleneck for certain people. So my request of you is if you ever get into this situation, register early, please. Don't turn up at the last minute. It's much easier. Then the next problem is that nobody is actually
13:05
really used to interacting online. I mean, you think it's easy. You've seen Zoom calls. You all just turn up, and you've got, you know, pictures of each other on it. If you try and do that with 800 people, Zoom doesn't really do autofocus very well. You can't see who's talking. And if you're just observing the conference, it just becomes
13:25
a mass of faces, and it's completely useless. So we obviously have standard issues like good webcam, good microphone. The number of people probably among you sitting in this
13:40
Dell system when you just open the laptop will do when you're chatting to each other. Believe me, it won't. The thing that audiences hate the most in online conferences is static, background noise, indistinctness, being unable to get the point across. If you don't have in the attendees who are going to interact good AV equipment, this is the thing
14:01
which actually makes the conference experience go down the fastest. This is about the only thing we got right, because we gave us speaker gifts designed to be delivered to anywhere in the world. A month before the conference, a complete gift pack of headset lights so you could actually see the face and a webcam you could put on the top of
14:22
your camera, a 1080p webcam so we were sure that we'd actually have a good view of the speakers. We made sure that they got to all of our nearly 100 speakers in the conference, and most of them didn't use them, which was a bit of a problem. We also found, as I've explained with Zoom, that raised hands don't work and having
14:42
a load of people on video who you have no idea what's going on doesn't work either. So the thing we pioneered for Plumbers is called video muting, which is where we try and instruct the attendees to, if you're just watching, don't turn your camera on. So pretty much it's a blank screen. The only people who turn their camera on are people who are talking or people who are waiting to talk. So it actually gives a good
15:04
visual clue that somebody wants to interject at this point. And we found it was actually a reasonably good way of facilitating interaction in an online conference. And it certainly solves the Zoom confusion where you just can't figure out who's talking to whom on your screen. And for the online conference, we did have about 900 attendees.
15:24
So you could imagine on a Zoom screen, everybody would probably get about five pixels. In 2020, BigBlueButton comes with an internal chat system, which means anybody logged into BigBlueButton can pop out a chat tab at the side, and they'll be able to chat with
15:41
anybody who's logged into the conference. So this is why, if you were doing streaming of the conference, you couldn't interact because that is a completely enclosed chat system. But we realized that people wanted to have conversations outside the discussion rooms as well, things like hallway tracks. So we actually adopted a sort of open-source project
16:03
called RocketChat. Now, the core of the project is fully open-sourced, but RocketChat themselves runs a more open-core model where there are certain enterprise upsells and everything. But this was just the one we came across. And the good thing for us is it doesn't require any integration because we'd already started to realize that the more integrations
16:22
you do, the more difficulty you have, the more strings and bailing wire stuff there is to go wrong at the crunch time when you put the conference on. So no integration looked like a good thing until we tried to run the conference. And then the problems were that chat was hard to read on the live stream because
16:41
we're streaming it out pretty much through YouTube. We actually, Gilunadi wrote a streaming module for BBB, and we just plugged it in as an additional attendee, and what it saw was what we streamed to YouTube. And the problem is that YouTube does dynamic compression of most of the stuff while it's streaming. And if you have a low bandwidth connection,
17:02
it will quite happily Greek all of the chat that's going on, and you just can't see what is going on on the chat. Whereas, you know, if it's sort of having a bandwidth problem with faces and you're just looking at lips moving, you can stand losing quite a few pixels, but if you're trying to follow chat, it's just impossible. And obviously,
17:24
streamers didn't have any credentials to rocket chat anyway because we based it on our LDAP. If you're not registered to the conference, you have no chat credentials. So they couldn't join the hallway track either, which they didn't like. This was the problem. And the third problem was we had to buy an enterprise license from rocket chat actually to get some of the notifications that we used for integration to work.
17:46
And the two chat systems were completely and fully disconnected. So you couldn't use your rocket chat login to look at the BBB chat and vice versa, which also was a bit balkanizing and a bit annoying for people. And so you can see that the thing I initially
18:02
portrayed as a huge advantage because we didn't have to worry about the integration was the thing that everybody found the biggest disadvantage about the conference because it wasn't integrated. So you always get this pressure of, and you know as conference organizers, the more integration you do, the more chance there is that something will go wrong. But if you don't do enough integration, your audience don't like it.
18:24
So you have to get the amount of integration right. And for several conferences, we actually struggled to do this. Every registered attendee got a matrix ID, and since we used email and confirmation number to log in, we figured we just use email address matrix as well.
18:45
But there's a slight problem here. Emails contain an at sign. Matrix does not like additional at signs in your ID. So we had to replace the at sign with a full stop, which I thought hardly be easy for our attendees. They're all computer geeks. They'll get this. They'll know email address for one server, email address but with at sign
19:03
substituted with dot for another server, complete disaster. You know, they just couldn't figure out which ID correctly to use for which. Eventually, we got the BBB server to use either form of the address, and we actually got the matrix server to translate using a JavaScript widget. If they typed in the email address, it would just fill
19:23
in the dot and everything would just work. We tried it without doing that for a couple of days and got a bucket full of complaints. I should also add that when we chose rocket chat in 2020, I wasn't very happy about the decision because I was a fledgling matrix
19:43
user and I was sure that matrix could handle the conference, and I thought it would actually be a good idea to do this. So I advocated even in 2020 that we should be using matrix. And finally, after all the difficulty with rocket chat in 2021, I got my wish. So they
20:01
told me, we're using matrix and you're in charge, which was fun until the first day came along and the matrix server with 900 people on it just fell over. And we had actually done scaling testing. I mean, I'm not naive. I know things can go wrong at scale that, you know, you don't see in your own lab. So we'd invited everybody
20:22
to come along for a town hall, but only 100 people showed up. And it turned out that most of the scaling problems we had were somewhere between 100 and 900. So we hadn't hit any of them in our little test. And I'd also tried to scale up the matrix
20:40
server. So our massive 16 CPU system that we were paying about $100 a month for, when you ran top on it, it was only using one CPU because I hadn't realized that matrix synapse, which was the server we chose, was single threaded. I mean, who in 2021 writes a single threaded web application? But the matrix people thread matrix by
21:05
using the back end web server through a sequence of proxies to actually do this. But that is not the default configuration. And so on the Monday of the conference, I was left there in my own little basement sweating away going, how the hell do I scale this thing? But fortunately, several people who attended
21:22
FOSDEM, which did use matrix before we did, also ran into this problem. And they were able to send me emails saying, yeah, we've got this guy you can talk to. He'll teach you how to do it. So fortunately, within about four hours of discovering the problem and realizing it didn't scale, we had a
21:40
roughly scaling matrix server working. And I managed to actually complete the scaling by the end of the first day. So for the second day, matrix was actually fully functional, fortunately, since it was my little baby. And I actually did a little blog post about how we got this to work, just so if anybody else ever runs into the problems, there are online resources to
22:02
go to to get yourselves out of it. One of the good things about using matrix was that even the free tier attendees, the streaming tier attendees, could now actually interact by chat. So anybody with their own matrix ID could log on to any of our matrix rooms, because they're all public, in
22:20
the same way that all the FOSDEM rooms are public. Every attendee still got their own separated matrix ID, but you only got logged on to the matrix ID if you actually popped up the side chat panel. You could also use your own separated matrix thing and use your own matrix ID as well if you wanted to. And it just depended how you
22:43
wanted to interact with the conference. Gui Lunardi was the one who actually modified the BBB server to, instead of opening its own chat tab, we just stole the iframe concept it uses for the blackboard, and we did a riot embedded,
23:00
which is done by some, the URL to the GitHub account is there. The integration, the riot embedded is not as good as running the electron type application, but trying to embed an electron application into a web form is just a nightmare and doesn't work. So it's a very cut down version. All it really does is text pictures
23:23
and reactions. It doesn't do any of these sort of complicated threading or replies or anything, but that's pretty much good enough for conference attendees. But you can see the primrose path we're treading down, because this is yet another integration into a BBB and matrix that is not done by either upstream.
23:45
But it was really useful, so the streamers could see the chat. If it was Greeked on their streaming screen, they could actually use a matrix client and still view the chat in real time without having to worry about video compression and reduction.
24:04
And so we thought, after we'd done this successfully for two years now, especially for one year with rocket chat, one year with matrix, it wouldn't be too difficult to move over to hybrid, which is probably really the point of this talk.
24:22
We already had all the remote infrastructure. By the way, to run this style of remote infrastructure for a conference the size of plumbers, so we're talking about 900 people, and we assume that for the in-person one, we'd probably be about 400 in-person, 500 online. It will cost you somewhere between $3,000 and $5,000 to actually set up and run.
24:42
It's not cheap to do the hosting of all of this stuff, especially because we had to host at scale. We needed one BBB server per each track room. We had about six track rooms. We had a few spare servers left over for hack rooms, and we had one for the hallway track. But the way we looked at it is, we already had all of the AV necessary for
25:04
actually just, this is the naive thought, which is where kernel developers go. You know, we've got AV in the room. It's got two cameras, and it's got a load of microphones. The room is just another single person attending the BBB chat. It'll be easy to do it like that, the naive way of thinking.
25:21
But it really, really is not that simple trying to get a hybrid conference running. The significant problem is that you have to integrate with in-room AV. And in-room AV, well, okay, so at ULB, you control it. But we were doing it in hotels. Hotels have their own local AV teams.
25:41
If you touch a piece of their equipment, they tend to go screaming to management. In order to get the integration done, you can't do it yourself as the conference organizers, the AV people, the audio people have to do it. And it's surprising the number of audio people who sort of sit behind those switcher boards at the back of conferences who've never even seen
26:03
a sort of online hybrid conference and have no idea how to set this up. So just plunking this sort of video display down in front of them and saying, hey, now you're in charge of audio and video, and we just want the conference to work, wasn't actually a winning strategy.
26:23
So we discovered rapidly, and unfortunately it was pretty much on the day of the conference that we had to train the AV people on how to use all of this stuff. And we were actually quite fortunate. Because we'd ordered the video-based AV, we actually had two sets of AV people.
26:41
So the hotel usually supplies audio people to plug in your projectors and your microphones into the hotel's audio. And then we had a separate AV company whose job was to manage the cameras and sort of cut the whole thing together and do the live streaming. So we actually had two AV teams effectively, and we were assuming that we could just co-opt the AV team who was doing the live
27:04
streaming because live streaming, running live streaming is not much different than running an online conference, right? Wrong. So we held meetings beforehand with them, and we talked to them about it. We couldn't hold meetings with the hotel audio guys, but
27:21
we assumed it would just be a plug for them, which it wasn't. And nobody really anticipated the issues that we actually had. Specifically, the AV setup of remote attendees is not obvious to somebody who's used to dealing with throwing microphones around a room. Because there is one extra wire that's coming off the Internet
27:42
over which a lot of attendee voices can come. And the first time they set up their AV patch panels, this wire just wasn't plugged in. The second time, it was bypassing all of the audio routing, which meant that the sound quality and the volume was chopping at high strength because it wasn't being attenuated through the usual filters.
28:03
I mean, this is the 101, how many screw ups can you have? We basically did them all on the first day. So let's go through a list of the problems. I even forgot to mention that between 2001 and 2002, we decided to upgrade the whole of the infrastructure.
28:23
We had specific things in BBB we wanted. Remember I told you we have two cameras. We have one at the back facing the speaker, one at the front facing the audience. The new version of BBB could actually handle multiple cameras with different views. So we figured that wouldn't it be really great if you want to see the audience,
28:43
so you just select the audience camera as a remote participant because you want to talk to them. Or let's say you're a remote speaker, you want to see the audience, but the rest of the people just want to see you. We'll do this multiple camera selection. It was going to be the cornerstone of our implementation of this. Problem was, nobody told the AV guys.
29:01
So the AV guys had come set up with a switcher for the two cameras, which is how they ran the streaming. Because obviously you can select one camera view or the other because the live streaming pretty much has one camera view out on it. And by the time we all sat together in the room, there was no way of unplugging this setup and redoing it such that we could plug in two video cameras.
29:22
Because it all has to be done from the AV control deck, and they physically didn't have another input because they were expecting to use a camera switch. So this is yet another screw up in planning. Third screw up was because we'd upgraded the newer BBB, they'd redone the, and this uses node react as the frame backbone.
29:42
They'd redone the way it was working. Riot embedded no longer worked as a pop out panel. We discovered this fortunately four days before the conference. So this was Guilinardi doing a very quick back end retrofit, trying to get the matrix chat that everybody was expecting to work to work. And fortunately by the time we did the conference, it was actually working.
30:02
But this was yet another squeaky bum issue where we tried to do everything at the last minute. The first day we had a lot of rooms where the room couldn't hear the remote attendees, this is the wire problem from the remote. Or the remote attendees couldn't hear the room because it's a two wire problem.
30:22
It's not just sound from the remote coming into the room, it's sound from the room going out to the remotes as well. So we had to have the sound correctly plugged into BBB. The streaming guys thought they were just plugging the sound into the streaming. So if you watch the live stream, it had the correct sound. But if you were on the BBB thing,
30:40
it was completely silent because they hadn't plugged the sound in. It's just yet another screw up that we didn't actually think to test out. So this is how it looks in the sort of conference hall as we were going around it. So this is the panel you can see. This is what BBB looks like pretty much in a classroom situation.
31:00
This is the presentation which you can actually minimize if you click this button. But obviously we thought that the AV people might be able to click the minimize button when the conversation's going on and then click it back again. No, it didn't really work. They didn't really want to do this. So we just kept everything up. This is the attendees panel, which is all scrolling, which is nice.
31:22
But if you think about it, that's not really what you want to see in the conference. The chat panel, if you click this, will pop out and come here. And the rest of this will really shrink, which is also not really what the attendees did. So what we should have done is have the chat panel down here and get rid of the attendee list or make it pop out only.
31:40
Well, so we put this on our to-do list for next time because it's sort of a nice to have, but it wasn't essential to getting the conference working. This is actually an excerpt from the RISC-V micro-conference. So Atish Patra is actually asking a remote question. You can see that he's asking it of a member of the audience, so we've switched to the audience camera view.
32:01
Because ideally there would have been two cameras here, one of the audience, one of the speaker, and he could have switched between them. But we screwed up and didn't get that working in time. The tab also includes shared notes, which goes to a back end on etherpads. So the remote audience can also see the shared notes.
32:21
And they can also click on the pop up. And these buttons here actually allow you to mute people who are not supposed to be speaking when they are. Cuz in a remote conference, you get lots of people who forget to mute their microphone and then have a conversation with their wife or their significant other. Then this happens quite often.
32:42
The way BBB is set up is that anybody can mute anybody, but we thought that might be a bad idea because it's sort of cantankerous kernel developers, you never quite know who they'll mute. And once you've muted somebody, you can't unmute them. They can only unmute themselves because this is a sort of, you can't be unmuting people while they're going to the bathroom or something.
33:01
It wouldn't be good on the conference. And we figured this might be a problem. So we restricted this functionality to administrators only. And then, nobody could figure out who the administrators were, so it became a bit of a problem. The video overlays work quite well. So, I mean, this is pretty much a blow up of a 1080p thing.
33:24
And you can see the greaking happening in the room. It's very difficult to make out individual attendees. But the sort of remote participants were reasonably visible to most people. So BBB, and especially, you can actually see the way it works with the video muting idea.
33:41
It's really, all you see are the people participating. Now, one of the other things that we'd actually tried to do in the room is the camera at the front facing the back actually had a pan and zoom facility, which the AV operators had assured us would allow them to zoom in on
34:01
whoever was speaking. But in the event, what we discovered is that only two of the AV operators out of our six rooms knew how to use this. So we did get panning and zooming in some of the rooms, but not in all of them. Again, this goes back to preparation and training. We will do better next time.
34:21
Yep. And it actually took us quite a while to get the audio routing right. You'd think that plugging two wires into an AV system wouldn't be that difficult, but in fact, it really was. And we finally figured out that what you had to say to the audio guy is,
34:40
here's a wire from what's effectively a microphone on the computer. The video guy will give it to you. You just plug it into your fader panel, remember you have it. And you give him a wire for the AV stream that also has to go into the BBB thing. And eventually, we got that to work. But obviously, we had other problems.
35:02
One of the other things that we'd really thought about was, when we get into the room, we'd actually like to have the chat and all the remote attendees on one monitor, and then we'd like to have the presentations on another. So we thought we'll have a two-projector setup, which sounds fine until you realize that we have rooms about as big as this.
35:21
And the two projectors were right over there and right over there. So somebody sitting over there can't see what's on that projector. Somebody sitting over there can't see what's on that projector. So the only way we could make it work was to display the same image on both projectors, which meant that two projectors was a complete waste of money.
35:41
And we should have gone for a single central one as well. So next year, we will either try with one or three just to see if we can actually get this right. Obviously, as I said, the two camera handling we thought would be the centerpiece of the technology. The reason why we'd upgraded BBB and caused all of the React JScript failures
36:03
on the matrix backend was to get this facility that we couldn't actually use. So that was not only sleepless nights for one person for a week to try and get it all to work, but complete screw up on the day when everything we thought we'd bought with that upgrade didn't function.
36:21
The other problem we had was because the AV people are actually doing the live streaming, and because of the way BBB works. There is no choice other than to actually run the master BB session that is showing up on both sides, both projectors, on their AV mux console.
36:42
And I had thought this might be a problem as well. So BBB is very heavily matrix, it's very heavily React JavaScript based. And certain browsers handle it better than other browsers. So it's mostly, it works with Firefox quite well now, but it is mostly optimized for Chromium.
37:02
So I'd actually already asked them, God, if we have to do this, what is the built-in browser of this tmux thing? And they said, it's Chrome. And I thought, God, it's gonna work. But the problem was that the mux platform might have used Chromium as an internal thing, but Chromium is open source.
37:23
So the guys who built the mux platform had just basically taken WebKit, gutted some of the JavaScripts, and hoped that that would be okay. And what it really meant was the threading functions of JavaScript didn't work very well. The result was that it kept dropping out of the conference. And obviously, since this is running the AV conference,
37:42
when it drops out, the in-room chat and the remote participants fully separate. And the problem is, there's no indication that this has happened from the console unless you happen to be watching very carefully. Because all you see is that a microphone disappears from that little icon in the top left that says, hey, you're the room.
38:00
And suddenly your microphone's gone. That's the only indication the AV people had they dropped out. Trying to train them to notice this and reconnect was a bit of a problem. So we do have automated reconnections on the back end. But we will try and integrate into BBB. But when we raised it on the list, they looked at us and said, well, how come you have this problem because nobody else has this problem?
38:23
If BBB kept on dropping people out of the conversation as a matter of course, people would have complained. This was a specific interaction due to the cut down browser and the TMUX interface that nobody had anticipated. Again, another screw up from not actually trying it beforehand.
38:42
So as I said, rejoin was manual assuming the AV guy noticed. One of the good things we did have was in-room shepherds who were committee members, so they could actually watch the on-screen panel. Every time the icon disappeared, I or one of the other shepherds could run to the back of the room and pat the AV guy on the shoulder and say,
39:00
you need to rejoin, please do this. So we could make it work with a lot of manual fussing behind the scenes, but it wasn't easy. So one of the other things that we realized is that hindsight is 20-20. I mean, they might want to run it from the AV console, but they don't have to run the whole thing.
39:21
What we could have done is have a separate laptop that actually ran all of the AV connections under the charge of, I won't say somebody competent, because it would have been one of me, the kernel developers, or somebody. But somebody who would pay more attention. And then what we could have done for the VMUX guys was to actually give them a VNC session or something like that into the panels that they were projecting over the streaming and
39:44
everything else. And that way, we would have had much more control in the room, and we'd have been running a full-fledged browser that wouldn't have had most of the connection problems that we had. But obviously, trying to sort that out on the day was pretty much impossible. Particularly, unfortunately, is this AVMUX platform is sort of Windows-based.
40:03
And Windows does not do VNC very well. And trying to get it to work with RDSP, which is the Windows native thing, didn't work very well. And we just, we were in Dublin, we didn't have any equipment. We had to organize the parties and deal with online conferences. We just couldn't get it to work. So we do know that for the future, we are going to try and
40:22
do a shareable console. And I need to leave time for questions. I have ten minutes left. We had a load of other problems as well. As you can see, this, so I'm, in order to try and tell you all of the difficulty about doing this, I'm talking up the problems.
40:44
From the audience's point of view, this conference was actually very successful. We got plaudits from most of the attendees that the interaction worked as they expected. They could have their one-on-one conversations, including between remote and local people, remote and remote people.
41:00
So everything appeared to just work for them, modulo the problems of not being able to hear the room and having to rejoin and having to tell people they'd lost AV and they needed to rejoin. They forgave us for all of that. So it was actually highly rated as an incredibly successful conference, but it was a bit like a swan with that serene sort of thing swimming over the surface and a load of committee members frantically paddling underneath,
41:24
trying to get all of this to stay up and running. Our other big screw-up was that we had a committee of ten people. But because this was pandemic, not all the committee could turn up in Dublin. And by the time we got to Dublin, we only had two local people
41:41
who actually understood the system and could deal with it. Surprisingly enough, they're the two people who are actually standing up on the platform now, taking the blame for all of this. But believe me, in a conference with six tracks and AV guys who don't know what they're doing, if you want a lot of exercise, please do it with two people.
42:01
But if you want to actually make it work seamlessly and vaguely competently, you need one person who understands the setup in each room. So next time we will try and do it with a minimum number of people as we have same number of track rooms. Again, it's not rocket science. It's perhaps something we should have seen ahead of time. But we thought it would be easy, and it wasn't.
42:24
So, to leave a bit of time for questions if you have them. Running conferences is actually hard, so just remember that when you go home from here. And volunteering for a conference is actually an act of public self-sacrifice that more of you should consider. Because honestly, you will find out that it is really hard.
42:44
Running online conferences is harder, especially if you try and integrate it yourselves, because there is actually a lot more to do. It is actually more difficult to run an online conference than it is to run an in-person conference, paradoxically. And this is mainly to do with the integration issues.
43:02
Everything that doesn't integrate well is either your fault or you're responsible for integrating it. And integrating web stuff, even in this magic day and age of sort of web scripting and operators and everything else, really doesn't work that well. And hybrid is really only for masochists. Only do a hybrid conference if you want to give yourself an untold amount of
43:24
pain that you can then come and complain to an audience like you about later. But if you really want to get it right, the key is training your AV people. Do not wait like we did to turn up on the day and say it'll be easy. You need a couple of training sessions a few months beforehand to try and
43:42
get all of the kinks sorted out. And having meetings and discussing it doesn't cut it. You need the AV people to set it all up and preferably have a few people fly out to see the setup, work with a few remote people so you actually check it out in all of its situations.
44:02
And obviously, you need to do this in advance. You don't do it on the day of the conference, which is what we try to do. Well, lesson learned. So with that, I think I have about five minutes for questions. This was actually all done in open source using HTML and CSS. That does make me a web developer.
44:21
Proud of it now, finally. Obviously, you can see that kernel developers don't quite do web as well as web developers. So if any web developers would like to join our program committee and become jointly responsible for next year's disaster, please come forward. The presentation is all online, and I've also put it up on the link in PentaBath.
44:41
So you should just be able to click on it. All of the links in the presentation that are highlighted are clickable if you want to see the URLs. And with that, I'll just say thank you. And Mike and I can now answer questions. You have a mic. We'll share this mic.
45:01
Any questions? Please speak loudly under the microphone. Yes, thank you. From what you said about the AV setups in the hotels, were there any
45:24
discussion to just ask hotels to move their stuff and have your own stuff brought in? Because to me, that seems after your talk to have been an easier solution to just say, you know, your AV guys and your AV is probably good, but we have really specialized needs. Could we just bring our own equipment in?
45:42
OK, so the question is, could we bring our own equipment in? And the answer is pretty much every hotel, no. Because remember, they do the AV routing as a very specialized sort of adaption for the conference rooms. They have a lot of really expensive speakers, sometimes in the ceilings, sometimes at the back of the room.
46:01
And they don't want any non-experts touching it. So what hotels usually have as a clause in their contract is a requirement that you use their own sound guys. And this clause is non-negotiable. We did try and negotiate with a couple of hotels where we said, because we had a few screw-ups even in the live streaming days where the hotel didn't do very well and we thought we
46:22
could do better. But it's always come back to us as an absolute no. It's actually very lucky for the ULB guys who are used to having students handle their equipment and don't seem to have as many hang-ups about it as hotels. So there are people here who can be trained on this equipment who can use it. But pretty much, when you're running a conference in a
46:42
hotel or in a professional conference hall that we've also been to, so a community convention center or something, you actually have to use their AV staff. You don't get a choice. And that means that you always have this extra component of people that you have to deal with. And they're usually people who don't really understand fully what your requirements are because it's very
47:02
difficult to communicate ahead of time. And these people are usually the people that you don't get to talk to until you turn up at the conference. So when you have your intermediate AV guys like we had and you have your meeting with them, you always have to make sure that they have enough knowledge and insight that they can do the knowledge transfer to the sound guys who are going to be coming in on the day
47:22
from the hotel. So yeah, it's a huge problem. And there's almost nothing you can do to get out of it besides sort of having a very friendly university who will let you play with their equipment because pretty much nobody in the professional circuit will let you play with their equipment. I have one question from the chat.
47:42
Was it worth it to record the audience? So the question was, was it worth it to record the audience? And the answer to that is actually very much yes. Because remember in a micro conference, we're stimulating discussions between arbitrary groups of people. So these would be people who are sitting in the
48:01
audience physically versus people who are sort of remote. And in order to have a correct two-way conversation, you actually have to be able to see the person on the other end. And so the whole point of the audience camera, at least when it worked with the panning and zooming, was that if you were having a remote to local conversation with the audience, the remote people could actually see the
48:23
local participant. And if we only had a camera facing the speaker, we'd either have to call every participant up to stand on the stage, which wastes time, or we'd just have to not have them be able to see the person they're talking to. And being able to see the person you're talking to is
48:40
very important for interactions. There are lots of visual cues that people get wrong. So I think having an audience camera is actually one of the good benefits of, or at least one of the benefits the plumbers did and one of the good insights we had to try and get an interactive conference running. So I wouldn't go back and do no audience camera, no.
49:02
I just wanted to get back to what you said about the ULB being willing to allow us to use their audio equipment. I just want to point out that this has a lot to do with the fact that we've been doing this for 20 years and they know us. It wasn't always the case. And if you're not forced them, or you're not working
49:23
with the ULB, this probably isn't the case. So could you repeat the, I got half the question. No, you. Yeah, it wasn't the question really so much as to point out that us being allowed to touch the audio equipment here by the ULB has a lot to do with the fact that we've been doing this for 20 years and they know us.
49:41
Okay, so the point being made is that ULB only allows the staff here to touch the equipment because they've been doing it a long time and they've built up trust. The way we work, we tend to go to a different hotel every time we hold plumbers in a different location. So pretty much, even if we could get the trust to build up with a hotel chain, we wouldn't be in the hotel
50:02
often enough to do it. But the important point to emphasize is that pretty much 95% of the time you will be forced to work with AV people who are supplied to you by whoever owns the equipment. You won't get a choice of who does it.
50:23
Okay. Is that it? Okay. Well, thank you very much indeed.