Drones, Virtual Reality and Multiplayer NES Games. The fun you can have with Pion WebRTC!
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 637 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/53657 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2021594 / 637
1
2
7
8
10
11
12
17
29
33
35
38
40
44
48
50
54
59
63
65
85
87
91
95
97
105
108
114
115
119
120
122
126
127
129
130
133
137
140
142
143
147
149
151
156
159
160
161
168
169
170
175
176
177
178
179
182
183
184
187
189
191
193
197
198
204
206
209
212
220
222
224
227
230
233
235
238
242
243
245
247
252
253
255
258
260
261
262
263
264
265
272
273
278
281
282
285
286
287
288
289
294
295
296
302
304
305
308
310
316
320
323
324
328
330
332
335
338
342
343
347
348
349
350
351
360
361
365
368
370
372
374
377
378
380
381
382
383
386
390
392
395
398
402
405
407
408
409
414
419
420
422
425
427
430
439
451
452
453
458
460
461
464
468
470
471
472
473
475
478
485
486
487
491
492
493
495
496
498
509
510
511
512
516
532
534
538
543
548
550
551
554
556
557
559
563
568
570
572
574
575
577
583
585
588
591
593
595
597
601
602
603
604
605
606
607
610
611
617
627
633
634
00:00
Game theoryExecution unitProjective planeOpen sourceProcess (computing)Goodness of fitComputer animation
00:42
VideoconferencingBinary fileWeb browserCommunications protocolErlang distributionJava appletServer (computing)Connected spaceInformation securityWeb browserPeer-to-peerCommunications protocolDemosceneMultiplicationInterprozesskommunikationOnline chatBinary codeVideo trackingMessage passingInheritance (object-oriented programming)Different (Kate Ryan album)ImplementationMultiplication signWebcamUniform boundedness principleFormal languageWebsiteSelf-organizationGoogolSoftwareState observerOperator (mathematics)Computer animation
02:27
Bit rateVideoconferencingBlock (periodic table)Higher-order logicIP addressRight angleSoftwareRoutingNeuroinformatikÜberlastkontrolleBit rateConfiguration spaceStreaming mediaPhysical systemAttribute grammarBlock (periodic table)Server (computing)Traffic reporting2 (number)File viewerConnected spaceMultiplication signRouter (computing)Single-precision floating-point formatInsertion lossMappingCommunications protocolVideoconferencingBand matrixTerm (mathematics)YouTubeSystem callDifferent (Kate Ryan album)Network topologyResultantData transmissionMusical ensembleMaxima and minimaDependent and independent variablesDataflowPublic key certificatePeer-to-peerMeasurementTunisRoundness (object)Computer animationProgram flowchart
06:28
BuildingCodeProgramming paradigmCartesian coordinate systemIP addressTraverse (surveying)TrailCodecMatching (graph theory)Functional (mathematics)2 (number)Dependent and independent variablesReverse engineeringStreaming mediaConnected spaceComputer animation
06:59
Letterpress printingWeb browserVideoconferencingFunctional (mathematics)ImplementationVideoconferencingStreaming mediaParameter (computer programming)Frame problemComputer configurationKeyboard shortcutMobile appWeb browserConnected spaceInterface (computing)Message passingInheritance (object-oriented programming)Process (computing)outputComputing platformTrailInformationCartesian coordinate systemBridging (networking)JSONXMLUML
08:33
Group actionElectronic mailing listProjective planeOpen sourceSpacetimeComputer animation
08:47
Scaling (geometry)First-person shooterProjective planeCommunications protocolInternetworkingSoftwareGame theoryPower (physics)VideoconferencingDefault (computer science)Cross-platformInformation securityInheritance (object-oriented programming)Traverse (surveying)Head-mounted displayNintendo Co. Ltd.Frame problemComputer animation
10:08
Point cloudGame theoryInstance (computer science)Right angleEmulatorUniform resource locatorSpeech synthesisVideoconferencingState of matterRemote procedure callFrame problemXML
10:39
Online chatWeb browserMultiplicationProjective planeYouTubeWeb 2.0Speech synthesisShared memoryInstance (computer science)Augmented realityRight angleOpen sourceInterface (computing)Videoconferencing
11:18
SpacetimeWeb browserTap (transformer)Client (computing)RoboticsBit rateRadio-frequency identificationÜberlastkontrolleFrame problemComputer animation
11:53
Standard deviationWeb browserAndroid (robot)Radical (chemistry)Communications protocoloutput
12:14
MorphingPoint cloudPlastikkarteCartesian coordinate systemGame theoryData managementLibrary (computing)Projective planeAxiom of choiceInternetworkingInstance (computer science)Streaming mediaBuildingPoint cloudRemote procedure call
12:52
Server (computing)Binary codeLevel (video gaming)Information securityWeb browserDemosceneWebsiteComputer simulationMappingPersonal digital assistantProcess (computing)Open setBlock (periodic table)Virtuelles privates NetzwerkInternetworkingPower (physics)Inheritance (object-oriented programming)Zoom lensOperator (mathematics)SpacetimeSinc functionLaptopProjective planeEngineering drawing
14:25
Computer fileWorld Wide Web ConsortiumComputer fileInternetworkingEmailClient (computing)FamilyWeb browserJSONXMLUML
14:45
PlanningVirtual realitySpacetimeDisk read-and-write headComputer animation
15:08
InternetworkingTwitterProjective planeMereologyBuildingMultiplication signServer (computing)QuicksortSoftware developerStreaming mediaSoftware repositoryLattice (order)Disk read-and-write headComputer animation
16:29
Element (mathematics)Computer animation
Transcript: English(auto-generated)
00:05
Hi, my name is Sean, and today we're gonna talk about Pion WebRTC. First off, before we actually talk about it, I wanna thank everyone who's involved with the project. Pion is a completely non-commercial, open-source project that, um, and we'll talk more about what it's doing,
00:21
but first off, I wanna thank, and these are all the names of the people that have contributed. It's a community project, and without these people, it wouldn't be possible. I'm super excited to see that we've now crossed over 300 contributors, and we've actually had 11 people in the community get jobs, so I'm really excited that how fast the community is growing, and if you're interested in being involved with open source,
00:41
Pion's a great place to be. So first off, before we even get into, like, what you can build with Pion and what it's all about, like, first let's talk about, like, what is WebRTC? WebRTC was originally designed as a protocol between browsers and servers, and it gives you end-to-end secure connection between peers, so that means that I can exchange
01:01
audio and video and data, and no one can see what I'm talking about. You can send multiple audio and video tracks. You can send your desktop and your webcam at the same time along with audio, and you can also do binary data, so that's super important if you're doing teleoperation or you wanna exchange chat messages or just metadata,
01:24
and it can be lost, it can be unordered, or it can act exactly like a WebSocket. It gives you that flexibility. And the great thing about WebRTC, it's available in a lot of places. So we have a Python implementation, we have a TypeScript, we have Go,
01:41
a Rust 1 is coming up, and then we have Google's implementation in C++. So we have a lot of implementations, and what's exciting is I think that WebRTC is slowly becoming maybe the best protocol to have inter-process communication between different languages, and it's not just, it doesn't require some PubSub server, it doesn't require them to be running in the same network,
02:01
and we'll talk more about, like, what that means later, but this is exciting me. And then also, outside of Python, I'm working on a book called WebRTC for the Curious, and it's on how WebRTC really works, not just the APIs we're gonna talk about today, and it's a deep dive on the protocol so you understand what's going on behind the scenes and the history of WebRTC,
02:21
and then also the great stuff like WebRTC in practice so you can understand debugging and teach you the sharp edges before you hit them yourself. So at high level, like, what does WebRTC solve? WebRTC lets you connect two users that have no public IP, and what that means in practice is you can have two computers that are in completely different networks.
02:42
On the left, let's imagine that's you and your home network. On the right is someone you want to talk to. Your private IPs don't route to each other. You can't say, I want to talk from 192.168 to a 10.5. You're in different subnets. It's never gonna work. But you do know each other's public IPs. So how can you talk to each other? How does this work?
03:01
WebRTC uses an attribute of networks called NAT traversal, and NAT traversal allows you to establish a temporary hole in your network by hitting a stun server or an outside host, and what that does is a NAT will actually give you a temporary hole that people can communicate with you back in just by sending an outbound packet.
03:22
So here we have a host inside this subnet that sends a single packet out to the stun server, and it responds. Now anyone can talk to you. Think about it almost like automatic port forwarding. Instead of having to go into your router and configure that this port resolves to this host, your WebRTC agent can create these temporary holes
03:42
and talk to anyone via them. The other thing that WebRTC gives you is mobility. So if you've ever used... tried to do streaming over TCP, either just data or video, you know that when you roam, you have problems with connectivity. You have to end the TCP connection. You have to start it up again.
04:01
You have to deal with congestion control and all these other things. The nice thing is WebRTC has that built right in with ICE restarts. So if I'm talking to someone on my phone and I'm having an audio call over my Wi-Fi, I can walk outside, do an ICE restart, and the connection starts up all over again. Even though my IP has changed, we don't have to renegotiate.
04:20
We don't have to re-exchange my IP. We don't have to do any of that. Like, it's built right into WebRTC, and the congestion control kicks in, and everything just works. And speaking of congestion control, if you've never heard this term before, a lot of people will think that you can just run a single, you know, bandwidth analyzer. So let's say you want to stream to YouTube or Twitch.
04:41
You think, okay, I'll sit down and measure my network, and that's how I'll configure OBS to stream. But in practice, that's not how real-world networks work. What if you're streaming, and then three more devices join a network, and you have congestion? So before you had 50 megabits per second available. Now you only have 50 divided by three because you have these additional devices. How does that work?
05:02
WebRTC has built-in congestion control that sends back reports and says, I got these many packets. I lost these many packets. This is my round-trip time. And the sender can actually adjust their bit rate live and give you the best experience possible. It's all about tuning. Like, what kind of video do I want to send? What bit rate do I have available?
05:21
And how do I tune that experience to give my viewers the best possible? WebRTC also has a solution for head-of-line blocking. So if you're not familiar with this problem, the issue is, what if you send some data that actually isn't that important, but then you're blocking stuff that is important that's happening now? So let's say you have a system
05:41
that's sending telemetry data. It's not that important, but you're actually sending valuable data. You don't want to block in the telemetry. So what WebRTC has is data channels, which uses a protocol called SCTP. And in that protocol, you can actually mark certain packets as, like, if this isn't delivered, I don't care.
06:00
And so instead of in a network that has a lot of packet loss or maybe you don't have, you have a bottleneck on bandwidth, instead of blocking, you mark a single packet with maps retransmit to zero, and you don't have that problem anymore. New data flows unblock and guarantee, and delivery is guaranteed through retransmissions. So on stream two and three,
06:21
you're guaranteed that packets will arrive. On stream one, you don't care. It's nice to get that data through, but if it doesn't arrive, it's not that big of a problem. So now that you kind of know, like, what WebRTC is, let's look at how do we actually build a WebRTC application in Go. WebRTC uses the offer-answer paradigm. So what that means is that you have two WebRTC agents,
06:42
let's say you and a friend. What happens is one of you makes an offer. That offer contains things like, these are the codecs I support, these are how many tracks I wish to send you, this is my IP address that I discovered via NAT traversal. You make an offer, the other side responds with an answer. So now that you have the offer and answer,
07:02
you have an established connection. You can create a data channel. A data channel is just one stream of data, and then the second function is an options argument that lets you specify, like, do I want lossy, do I care about packet loss, and other things like that. And you create that data channel, and then you set an onOpen handler.
07:21
When the data channel opens, you send some text. And what's great about this is this works between so many platforms. So I can set up a data channel connection between a browser. I can set it up between a Go process. I can set it up between a mobile application. You have WebRTC that's supported on Android, iOS.
07:40
We have a Rust implementation coming up. This is what's super exciting to me, is these data channels, I think, are the best way to exchange data that exists right now. And then on the other side is just receiving data channel messages. You set a handler for onDataChannel, and then you set a handler for onMessage. We also provide a more Go-like interface
08:01
that feels like an I.O. reader-writer, but this is the idiomatic WebRTC implementation. And another cool thing is if you write your implementation in Go, we actually provide Wasm bindings as well. So you can build just as you expect in any Wasm application, and it spits out
08:20
and it works right in the browser. Sending video is easy as well. You create a new track, you add frames to that track, and you send them. On the other side, you send an onTrack handler, and you read from them, and that's it. So what kind of things could you build with Piner? Like, what are the things people are building today? These are some of the open-source projects that excite me the most in this space,
08:41
and there's a lot more out there than just this. So if you get a chance, check out Awesome Pion. It's a list of all the things that people are building. This is really a cool one, where a user has a Nintendo Switch that they're sending the video frames out, and then they're sending it to their VR headset. So this is a testament to the fact that you have WebRTC Sport in an Oculus headset
09:00
that you can send video frames over the network, and it's low latency enough that I can sit with a Nintendo Switch in my hand and play the game, but it still feels responsive enough that it's still worth playing a game. So right here, we're not playing something fast enough like a first-person shooter, but it does show, like, how immersive and of an experience you can build. It's a cross-platform protocol,
09:21
and you have this kind of flexibility. This is a Go project called Kerberos, and what it's doing is it's making existing security cameras available over the internet in a secure way. So RTSP, by default, doesn't have security,
09:40
so you probably shouldn't be streaming it over the internet, but thanks to WebRTC, you can run a WebRTC agent in the network with a security camera and then watch it over the internet. You don't have to put these cameras on the public internet because you have NAT traversal. You don't have to worry about security because WebRTC is secure by default. It's actually mandatory.
10:00
So this is a great project, and it's super easy to deploy, and it's another example of, like, what kind of power WebRTC can bring to the Go ecosystem. Cloud Retro lets you play old NES games right over the internet, and how it works is it's running an NES emulator on a remote AWS instance, and it's shipping the video frames back to you,
10:21
and then you're controlling the emulator via the data channels that we talked about earlier, and because you're running this emulator in a shared location, you can play games with a friend. So you can play multiplayer games. You can save your game state. You can do lots of interesting things thanks to it running in a central location.
10:40
And speaking of running stuff in a central location, this is a really exciting project that I think took off during the whole work-from-home and pandemic is the ability to run a web browser and not have to... and run that web browser in an AWS instance and share it with other people. So you access this web browser, and multiple people can control it at once.
11:01
I can watch a YouTube video, and multiple people can click around and interface. We have chat on the right, and you can do things like browse documentation together. You could work on a project. You could shop together. It's this browser. It's this co-browsing experience that works for everyone. Completely open source, and you can go check it out at that URL. Telago kind of taps into the robotics space.
11:23
So it hasn't been published yet, but there's a couple really interesting companies that are using Pion to do teleoperation with WebRTC. So you're sending robots the commands over the data channel, and then you're receiving video frames back via WebRTC. And it has all these cool things like congestion control.
11:40
So as the robot moves in and out, it can increase or decrease the bit rate depending on what's available. It's ubiquitous, so now I can control the robot via my phone, I can control it through a browser, or I can have a native client. And then WebRTC also lets me take conferencing right into my terminal. So WebRTC isn't just a browser protocol.
12:03
Isn't just, like, an Android or iOS. Here I am chatting with someone in their browser, but I'm in a terminal. WebRTC is a standardized protocol that works in lots of places. The creator of Cloud Game then created a subsequent project called Cloud Morph, where now you can run Y applications
12:22
and stream them over the Internet. So you can run, you know, Diablo 2. You can run CAD on a remote GPU instance in AWS. It opens up a lot of these possibilities, and it's all simple and scriptable. And, you know, the great thing about Go is how easy it is to deploy these things. There's no building C++. There's no dealing with, you know,
12:42
pulling in a bunch of libraries via the package manager of choice. It all just works. It's a pleasure to use this, and the latency is great. It's a fantastic project. I think WebRTC also has a chance to really revolutionize how people do things in the op space. So a lot of smaller companies,
13:00
they'll have a jump box where you SSH into one host that's on the public Internet, and then you'll have SSH into host behind that. You don't have to do that anymore with WebRTC. Because of that NAT traversal, that laptop can access the server, and it just works. You don't have to put the server on the public Internet. It just, they connect right away. I think that's super powerful.
13:20
It's great for security. It reduces the amount of ops burden of putting those servers, you know, managing mappings and names and stuff like that. Open source, and I think you can build some really cool stuff with this. You don't have to worry about running VPNs. I think, and then you also can access right from your browser. There's no reason that it has to be two Go processes.
13:42
You could SSH in or simulate SSH over WebRTC. Snowflake was one of the original Pion projects that kind of, like, brought more attention to Pion, and it runs, uses the data channel to do censorship circumvention. So if you have a user that, over data channels,
14:02
they can request, hey, can you please send me this website that's blocked by my ISP or by my government? It'll download it for you and send it back over WebRTC. And the great thing here is you don't have to download a binary. That binary is probably blocked by the template already, so you can't download it, but you can't block WebRTC since it's used by so many important things
14:21
like Hangouts and Zoom and other things like that. WebWormhole lets you exchange files over the Internet, so you can, you have a Go client and you have a browser client, or you can do GoToGo. So you don't have to send a file via email and then have it be decrypted and have your, worry about your mail host accessing
14:41
and other things like that. Like, it's peer-to-peer, and the only person that can decrypt it is on the two ends. And then I think Pion's also bringing a lot of interesting things to the VR space. So here we have a VR experience where the user is moving through the virtual space and their head is actually imposed on their avatar as they're moving through the space,
15:00
and you have these interesting things like spatial audio and recreating all these experiences that we can't have now because of the work from home and COVID. Project Lightspeed is a brand-new project that lets you stream OBS to a public server in sub-second time. So instead of having to set up an RTMP server and worrying about HLS, it just does OBS via WebRTC,
15:23
and then anyone can view it. It's super powerful because you don't have to worry about transcoding. It's really easy to deploy, thanks to Go, and a really bright future head for this project. So there's a lot more projects. GitHub.com and go to our awesome Pion repo. We want to share your project, so if you're building something cool,
15:40
we'd love to promote it for you. I think there's a lot of great career opportunities that also come from this. I've had developers that built this interesting project. They got hired out of it. They put it on their resume, so if you're, like, looking to get involved with something, we'd love to have you. So, you know, come get involved. Join our Slack channel and gain deep WebRTC knowledge, and it's also a fun challenge where you pick the goals.
16:02
You're on your own timelines. If you want to build something, like, you get to own it and build it all the way through, which is a really welcome challenge for a lot of developers. So here is all the places you can find us under github.com slash pion. Grab an issue, reach out to me directly on Slack, and I'd love to help you get started. It's completely non-commercial.
16:21
And it's a great place to get involved. So we'd love to have you and hope you learned something interesting from this talk. Thanks.