jxr in /engine/ - coding in WebXR on a plane
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 542 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/61866 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2023465 / 542
2
5
10
14
15
16
22
24
27
29
31
36
43
48
56
63
74
78
83
87
89
95
96
99
104
106
107
117
119
121
122
125
126
128
130
132
134
135
136
141
143
146
148
152
155
157
159
161
165
166
168
170
173
176
180
181
185
191
194
196
197
198
199
206
207
209
210
211
212
216
219
220
227
228
229
231
232
233
236
250
252
256
258
260
263
264
267
271
273
275
276
278
282
286
292
293
298
299
300
302
312
316
321
322
324
339
341
342
343
344
351
352
354
355
356
357
359
369
370
372
373
376
378
379
380
382
383
387
390
394
395
401
405
406
410
411
413
415
416
421
426
430
437
438
440
441
443
444
445
446
448
449
450
451
458
464
468
472
475
476
479
481
493
494
498
499
502
509
513
516
517
520
522
524
525
531
534
535
537
538
541
00:00
CodeFirefox <Programm>Computer animation
00:20
Firefox <Programm>Broadcast programmingConvex hullExecution unitComputer animation
01:00
PixelComputer-generated imageryPrice indexContent (media)LaceCodePresentation of a groupPeg solitaireComputer programmingNeuroinformatikBitoutputComputer scienceComputer animation
01:39
TwitterMessage passingUser profileService (economics)Data modelHypermediaTrailJava appletScripting languageVideoconferencingLattice (order)Computer hardwareProof theoryFormal languageBroadcast programmingEvent horizonComputer hardwarePlanningCodeInformation technology consultingHead-mounted displayLengthComputer animation
02:40
Content (media)WikiInterface (computing)Event horizonEmailArithmetic meanRevision controlWikiIntegrated development environmentVirtual realitySpacetimeComputer animation
03:58
Point cloudDemo (music)Control flowSource codeComputing platformVirtual realityTwitterGreatest elementMacro (computer science)Linear mapData modelData managementGraphics tabletComponent-based software engineeringProcess (computing)SmoothingDialectGraphical user interfacePresentation of a groupContent (media)Maxima and minimaFirefox <Programm>Thread (computing)Message passingPrototypeKeyboard shortcutUser profileVideoconferencingScale (map)Interactive televisionRotationSign (mathematics)CodeInformation privacyHTTP cookieMobile appNP-hardWage labourGreen's functionWordTwin primeWide area networkMeasurementFloating pointSpeech synthesisObservational studyLocal GroupGroup actionAverageWikiView (database)Reading (process)ComputerComputerBit rateLattice (order)Standard deviationWeb pageOpen sourceCASE <Informatik>Point (geometry)SpacetimeLink (knot theory)PlanningFunctional (mathematics)Different (Kate Ryan album)LaptopMereologyComputer programmingHacker (term)Online chatBitSoftware development kitTouchscreenOffice suitePresentation of a groupKeyboard shortcutParameter (computer programming)WordInterface (computing)Computer animation
08:00
User profileMessage passingTwitterCodeFirefox <Programm>Process (computing)Thread (computing)Price indexHill differential equationNormed vector spaceContent (media)Presentation of a groupEquivalence relationMeta elementKeyboard shortcutVideo game consoleParsingError messageVolumeoutputInterior (topology)Convex hullInformation privacyComputer-generated imageryVideoconferencingEscape characterPressureWeb pageCodeProgramming languageIntegrated development environmentSpherePresentation of a groupVolumenvisualisierungKeyboard shortcutCASE <Informatik>String (computer science)Escape characterEntire functionContent (media)Computer animation
10:49
BuildingComputer programmingFormal languageElement (mathematics)Interpreter (computing)Token ringVideoconferencingPay televisionMereologyAbstract syntax treeMathematical analysisFirefox <Programm>Finite element methodCodeIntegrated development environmentProgramming languageLimit (category theory)BitWeb browserExterior algebraComputer programming
12:06
CodeContent (media)Presentation of a groupFirefox <Programm>Limit (category theory)Execution unitConvex hullSource codeGame theoryFunction (mathematics)Process (computing)RecursionTerm (mathematics)AbstractionTable (information)Point (geometry)Sheaf (mathematics)ComputerGroup actionElement (mathematics)TwitterGoogolComputer animation
12:24
Table (information)Content (media)Source codeProcess (computing)Function (mathematics)RecursionSheaf (mathematics)Point (geometry)Term (mathematics)ComputerAbstractionCodeReading (process)AreaUser profileMessage passingComputer iconTwitterFirefox <Programm>Collatz conjectureCodeComputer programmingFormal languageLine (geometry)Endliche ModelltheorieContrast (vision)Group actionMultiplication signPrimitive (album)Revision controlBasis <Mathematik>Programming languageComputer animation
14:10
TrailNormed vector spaceFirefox <Programm>Presentation of a groupContent (media)CodeBranch (computer science)Function (mathematics)Context awarenessLatent heatKeyboard shortcutIntegrated development environmentElectric currentString (computer science)Attribute grammarFundamental theorem of algebraProgrammable read-only memoryFormal languagePoint (geometry)Link (knot theory)HTTP cookieComputer programmingInternetworkingWeb pageMachine codeTwitterAugmented realitySoftware developerText editorInstance (computer science)Message passingUser profileInstallation artProgramming languageGraph (mathematics)TesselationRepository (publishing)Process (computing)Integrated development environmentElectronic visual displayBitLibrary (computing)Endliche ModelltheorieComputer animation
15:52
File viewerBroadcast programmingBeta functionPresentation of a groupView (database)Java appletScripting languageTwitterStatisticsDigital photographyCodeWikiPoint (geometry)Graph (mathematics)Web pageTesselationElectronic visual displayComputer animation
16:40
Message passingCodeSoftware developerAugmented realityRepository (publishing)Installation artInstance (computer science)Entire functionUser profileThread (computing)TwitterText editorMeta elementBlock (periodic table)Latent heatComputer animation
17:22
Perspective (visual)CodeMoment (mathematics)File viewerView (database)Presentation of a groupJava appletScripting languageBroadcast programmingBeta functionTwitterMessage passingThread (computing)User profileTouchscreenMathematicsIntegrated development environmentCodeComputer animation
17:45
CodeKeyboard shortcutBlock (periodic table)View (database)Point (geometry)TouchscreenComputer programmingInteractive televisionComputer animation
18:25
PrototypeTwitterFirefox <Programm>CodeMessage passingView (database)StatisticsThread (computing)User profileLogicPhysicsContext awarenessBranch (computer science)Keyboard shortcutPhysical systemRule of inferenceCausalityTime domainSystem programmingComputer programmingMathematical analysisPrice indexLattice (order)CodeBlock (periodic table)InformationSet (mathematics)Field (computer science)Atomic numberFile archiverSelf-organizationRule of inferenceObservational studyTwitterComputer animation
20:08
CodeContent (media)Maxima and minimaSoftwarePerspective (visual)Software maintenanceShape (magazine)Firefox <Programm>Server (computing)Sanitary sewerWebsiteFirewall (computing)Meta elementProgramming languageSet (mathematics)Computer animation
21:10
Message passingMobile WebCodeBoundary value problemDivisorView (database)Personal digital assistantKeyboard shortcutHead-mounted displayTwitterMixed realityUser profileMeta elementExterior algebraBlock (periodic table)Point (geometry)FacebookAssociative propertyComputer animation
21:47
Server (computing)Sanitary sewerDivision (mathematics)Computer-generated imageryPixelCodeInterface (computing)Point (geometry)Computer programmingComputer animation
22:16
Software maintenanceSoftwareMUDMessage passingBoundary value problemCompact spaceTerm (mathematics)DivisorKeyboard shortcutPersonal digital assistantHead-mounted displayMobile WebPixelComputer-generated imageryFirefox <Programm>Server (computing)CodeContent (media)BuildingIncidence algebraMultiplication signBlock (periodic table)CodeInteractive televisionRadiusMetreArithmetic meanComputer animation
23:37
Division (mathematics)Computer-generated imageryPixelContent (media)Web pageComputer animation
24:02
Program flowchart
Transcript: English(auto-generated)
00:05
So, this is more of an entertaining talk, let's say. It's a lot less technical, probably a lot weirder, I would say. And hopefully I'll go fast enough so that you can have all the crazy idea or questions
00:24
at the end. And even a little demo, I have some hardware with me. If you didn't try VR, at least I think today you should try this if you feel like it. So that will be, like, just there where it's fresh. So I will...
00:41
So those are my slides, it's a JSON file, yes, why not. I will start by saying that I gave a couple of workshops to kids, actually, on discovering VR.
01:00
And that was pretty nice. I thought initially I would give workshops and give, you know, a roller coaster. Some kids asked me about murdering clowns on top of a roller coaster. I did not prepare this, I admit. But that was, yeah, interesting different inputs.
01:21
I think it's really interesting also to consider teaching computer science programming because I was saying there my pinewash, like, there is a computer everywhere. That's just a computer. So it might sound weird, might be a little bit outside of people's comfort zone, but in the end you can program a computer, you can program that.
01:42
So I will introduce myself a bit first. So my name is Fabien, Fabien Bénetout, Utopia. I work mostly at the European Parliament as a WebXR consultant. So I just do VR and AR on the Web, that's it. And I do it as a prototypist.
02:02
So meaning all the tips you had or the suggestion you had at the beginning of the day on TypeScript, on quality code, I don't know any of that. As I was saying before earlier, chatting with somebody at the entrance, if my code runs during the duration of the demo, I'm happy.
02:21
The next day, yeah, it's a bonus. But yeah, that's the length mostly. And I gave a couple of talks here at first them on connecting exotic hardware like the watch or IOT, everything, of course, using JavaScript. And this one will be on how to do this on a plane in a VR headset.
02:44
So my motivation behind this is I have a bunch of notes, I have a wiki. That's the wiki and I have a. You can still hear me, the 2D visualization of it, but it feels flat, like I don't want to have my notes in my skin or I don't want to have my notes
03:02
behind the screen, I want them everywhere so that I can play with it. Like I organize the space. So for this, I already had a couple of versions. I use Mozilla hubs. So a little show of hands, how many of you tried VR at all? OK, I would say two third, half, two third.
03:21
How many of you tried Mozilla hubs? OK, five, six, so it's a social VR experiment. It's a pretty good solution, to be honest, works quite well. And you can do this meaning you can be in of your headset with your hands moving. You can be on laptop, on a mobile phone,
03:43
and you can see this shared virtual space. It's using 3GS for the 3D environment. And you can have PDF, you can have your webcam, you can take screenshots. You can honestly do pretty much most of what one would want to do. So I did quite a few hacks or modification of it.
04:05
It's obviously open source. That's the interesting part. And but it's not enough. So you can do a bunch of hacks. So explain a little bit how, for example, I turn on and off the lights in my office while I'm in VR.
04:20
Director kits on how to record with traveling, bunch of different things to the point that I did my little toolkit on how to customize. Yeah, that's not going to work here. So you will have the link on the page. So imagine a bunch of functions related to hubs to do all this.
04:44
So I really think basically here the point is hubs is a good solution. It works quite well if you want to do just social VR. The thing is, first of all, it doesn't work offline. And I started after the end of the pandemic to start to fly again to go see family.
05:03
And I thought that's one of the best use case for VR. Like, I don't like being on a plane. I don't know if you do, but I think if you're the pilot, it's fine. But otherwise, as a passenger, it sucks. And one of the beauty of those things is you put the headset on and you're basically somewhere else. But I don't want to be in VR on a plane
05:23
and do a roller coaster or something like this. I want to code. I want to be there and be able to do something that I have agency over. I want to build stuff. I want to create things. And I don't want to be like this. I want to feel like I'm somewhere else. Then I started to to build on that.
05:43
Including, let's say, managing the history in my browser, I would take a snapshot of the different pages that I visited, be able to organize them. The thing is, as you might be able to see, this doesn't look at all like Mozilla Hubs. So Mozilla Hubs is amazing, but it's a huge stack, actually.
06:02
And it's a stack that depends on AWS, which I obviously don't have on my plane or the plane I'm flying with. So then I started to rebuild it from scratch, basically. One warning also, when you start to hear programming in VR, I had a discussion with somebody just yesterday about it.
06:23
It is it might be. Productive, I would say in at some point, but that's not my point. I don't find that interesting, like I have a really nice desk at home. It moves and all and I have my ergonomic keyboard and I can move my 4K screen. I like it, but I don't have that on the plane.
06:43
So the point is being able to move in space when I have the space and if I don't, to be able to use the space around me in a really compact manner. So the point is, this whole argument is like, don't look at this presentation thinking, is it to replace the way I work today? That's not the point. It might be if you enter it, but it's more exploring
07:02
what could be interesting in terms of interfaces. So, yeah, word per minute. If you're trying to sit next to me with your laptop and we do a competition of how efficient you are, you're going to win. I mean, I hope for you, but that's not the point.
07:21
And I was able, actually, on the plane to make friends because the person next to me was like, what's this guy doing with his keyboard on the on the tray? Plus, there was no USB-C on it. So I need to plug the adapter, put it on top of the device and then put it was not wireless back then.
07:41
So now I'm pretty happy I just received this little thing. So it's a Bluetooth keyboard and it's you can hear mechanical one. That's way more efficient. Back then, I didn't have this. So, yeah, that was. A good excuse to to meet someone and chat about what does it actually mean to to program in VR?
08:04
The next step being actually tinkering with it. So I went from having the history of my browsing behavior, let's say, to moving content and moving content by just pinching basically a page.
08:23
So that's a render of a page. That's not a web page itself. What hopefully I don't know if you can see all the way there, but basically with my right hand, I was I'm able to move things around. And with my left hand, I'm able to execute code. So that's, let's say, the main trick. There was one thing you need to think about that presentation.
08:44
It's about moving things around freely or naturally. But also you will see the so I move things. The red sphere there. And then if I pinch on that piece of code, the sphere there is blue. So it's not just moving things around, but it starts to manipulate text,
09:03
which takes this code and then code can change the environment you're inside of. And the one trick to do this, I will. I did away, but
09:21
you can guess. How do you execute code in JavaScript? The one thing you're not supposed to use evil because it's evil. It's I think that's the one use case when you can use it, when your string of text is actually code and you're not running a bank or something like this, like this is my code running on my headset.
09:41
I don't have anything that I want except to break, let's say, but even then it's executing locally. So that's the one reason in my opinion to use it. So the idea behind it is like, oh, I can manipulate text. That text can be code. I can change that text directly while I'm in the headset,
10:04
either pinching it like single letters or with a keyboard to be more efficient. And then I can execute it, which is going to change the whole environment I'm inside of. So the trick for it was basically
10:22
what I like to think of the escape escape valve, which is to say that, yeah, I could redesign an entire programming language. I could make it a programming language for VR, but I can also rely on one I already know. And that's already has an engine in it. So namely, JavaScript.
10:40
So that's a little, let's say, trick. So I'll show a couple of examples. But that was an excuse not just to manipulate text and this code, but then to start to make my own programming language. So how many of us here have made their own programming language? Yeah, I see.
11:00
There are very few hands that are like this. Most hands are like that. I'm like that, too. I don't think honestly I feel a little bit maybe not a shame, but I'm not proud of it either. What I'm trying to convey is the endeavor. Trying to do it is super interesting. I don't think I've learned as much recently as trying to do this. I also recommend if you check the slides or the presentation after
11:24
don't learn my programming language. I don't think it's a good one. It's a very interesting one for me by building it, by seeing the limitations of my understanding of what the programming language is. But so this why do I recommend this course specifically? It's because he's not going to let's say let's do an alternative to C,
11:44
which for most of us would be too far and not practical. He's basically saying, well, JavaScript is a language. The browser will notice an environment you can already work with. You can transpile, you can use Babel, and that's enough, let's say, to get the foundation of what a programming language is.
12:02
So I really recommend this kind of course, if you're considering programming, making your own programming language. And also, I don't know how many of you are familiar with with this book.
12:20
Yeah, I don't think I can go back to you. So, yeah, if you if you ever try to learn, I think the foundation of programming and programming languages, usually you often do it with LISP, which works well because it's so compact that you can keep it all in your mind, basically.
12:44
But again, most of us are not familiar with LISP and even if we are, then it's not what we work with on a daily basis. So it's ICP, but the JavaScript version has been out for a couple of years now. So if you want to learn, it's at the foundation of a programming language while being familiar with the language itself.
13:03
I really recommend it. It's quite taking, but I think it's really it's a good investment. So then I'll show a couple of the features of the environment, but I won't show all of them go too deep in there.
13:23
But then what's interesting is you don't have just one line of code, but then you start to have groups of lines of code or history, the stack of which you executed, of course, because you're in the R code is not everything. So, for example, some of the code is about loading a 3D model.
13:41
So if you pinch it again, you execute that line of code, you get a 3D model out of it, and then you can do it a couple of more times. You get I see that the contrast is not high there, but you get the history of what you just executed. Then you can save it and execute it again. So you start to have the same primitives you're used to.
14:05
And then you can do shortcuts, you can visualize the groupings. So for the people who twisted their hand instead of raising it,
14:26
this is also where I'm at in the sense that I don't actually know if it's a fair thing to call this a programming language. Is it just like a bunch of utils, a bunch of shortcuts? I'm not sure. Again, what I find interesting in that process of making a programming language is, yeah, it's up to you.
14:42
It's up to what you find interesting. It's up to what you find efficient. So, yeah, I start trying to document it, stack back a bit. But all what it prompts me to think of is really, really interesting, really valuable. And also, if you go down that route, there is a subreddit for it.
15:04
It's a subreddit for pretty much anything. But, yeah, for R slash programming languages where you can ask simple questions like this, like what is the most interesting aspect of it?
15:28
So, of course, I had to go meta. So I display my issues from my repository in my VR environment so that I can move them like as if I was on a wall. I can display 3D models from, let's say, a library of 3D models.
15:43
This way I can organize it like tiles and then start to make the 3D environment to work out enough. I can do honestly this part, the graphs. I mostly did because of finance during lunch. I was like, this is annoying. Everybody find graph sexy. It is looking good, but I'm not sure what's the point.
16:00
OK, I'll just make a graph. So those are actually the pages of the wiki where I'm taking the notes out of the display there. And they're all manipulable. Taking screenshots also in VR, I think that's a great way to document if you're doing a process. Of course, you can have the code out of it. You can save it back to your wiki, for example.
16:22
But the movement, how things are going to be organized and how you want to, let's say, organize the tiles to make the 3D world. You want to tell a story to someone who might not have no want to use a headset. Then just taking a screenshot, capturing it and then sending it elsewhere. It's pretty efficient way to do it.
16:49
That was just this morning. So there's the same thing of managing issues or displaying the issues from the repository live. I can again grab them, move them in places, but using Swagger.
17:01
So if you're not familiar with Swagger, it's a meta API. It's an API to manage all APIs so that once you have access to when you load using Swagger, yeah, I can load from GitHub, whatever is going to have a Swagger specification. And that means making them as, for example, manipulable blocks.
17:23
It's federated, so it's social. But if you have your own server, then you can connect to it. And we can be friends. We can exchange code also. So I can move the code in my environment and you can receive it and execute it. One port that is kind of funny is snapping.
17:42
So those are blocks. You cannot hear it. So we'll have to trust me. But when the blocks are close enough, you hear them snap. The points of those blocks, of course, or blocks rather are not just blocks is you can attach code on it. So then you can manipulate them literally like Lego block.
18:02
That's again, the point of the interactions that become interesting. Yes, you can have a keyboard and the keyboard works well. But if you just do the keyboard, yeah, you could just use a bigger screen view. It's not that interesting for you if you can have both programming with a keyboard or naturally manipulating or simply,
18:20
let's say, directly manipulating with your hand. I think that become, again, interesting. So again, you need to imagine those are not just blocks or blocks of code. It's called, let's say, for Swagger so that you can get the information from an API or execute through that API.
18:41
So why also do I do all this with the blocks? It's because I think that's a great way to learn. So you can imagine this, the blocks for running code that is abstract, let's say list things, but you can use that for chemistry. Let's say it's a C and it's a C or it's an H, H and an O
19:00
and you bring them together, they snap when it makes a molecule. I think the same kind of things apply to pretty much any field of study. So physics, chemistry, sociology, like you have a set of atoms and a set of rules. If you get them to have a certain organization specialized together, then you can learn this field,
19:22
hopefully in a more embodied way. Is that going to be it?
19:46
I have a Twitter archive because it's not working properly, but not for recent things.
20:25
So that's fine. All this, I think it's pretty interesting. The one thing I'm even more awkward with rather than calling the thing I've done my programming language is this. So that headset is an Oculus Quest or a meta quest. I don't have a meta account.
20:42
I don't think it's I don't think it's a. Yeah, I don't think civilians, capitalism, it's a good business model, let's say, as a society. But it works really well. So that's pretty awkward. I really don't think it's such a good thing. And I, I called it an adversarial dependency
21:00
because it helps me to build more, explore more than on something I don't want to rely on because it's not aligned with my set of values. So I still show all this because. Yeah, you won't see the photo, but there are a couple of headsets like the Lynx that should be there in a couple of weeks. The point is, don't associate VR
21:23
with just meta or Facebook. It is a convenient one, but there are alternatives coming out. There is also a similar VR. It's it's not the only way to do VR. So adversarial dependency in the sense that, yes, you can rely on it temporarily as long as you can have it just as a replaceable block.
21:41
And as long as you can swap it whenever you get something that's more than with your value, that is much better. Um, but that's it. Yeah, the whole point of this is building your own scaffolding, being a programming language, being a tool chain, using VR, because I find that to be the most interesting interface.
22:00
But yeah, that would be my recommendation for you. Just make your interface, make your scaffolding using VR or not. And if you want to try the VR one, I'll be around with it. So thank you.
22:23
We have time for a couple of questions. Do you have any questions?
22:47
I really like the VR experience in the in the play. But have you thought about the incidents that may occur like hitting my neighbors, punching him?
23:00
Thank you. Yeah. So I don't recommend punching anybody, but you're not good. So you you define the interactions however you want. So basically, if you put things, let's say a 3D object, either a block of code or a hat, whatever, a meter away from you, you're going to punch your neighbor.
23:21
So but because now you have access to the code in there, you can just say, oh, I bring, let's say, everything within a 50 centimeter radius. And you can just do it on the spot. Again, it's using WebXR. So it's VR and AR on the web, meaning there is no like unity build. There is nothing there.
23:40
Everything is done directly in the headset. You can put your Node.js server, your even your Python server, whatever you want to run. So it means you can refresh the page and then you have your new content and your way to handle it. So either you do it without even having to reload, but you can fix it on the fly, let's say.