The year of the virtual Linux desktop
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 490 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/47444 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Virtual realityRoundness (object)VirtualizationComputer virusOpen sourceMultiplication signTrailNormal (geometry)Positional notationLimit (category theory)State of matterVideo gameComputer animation
01:12
TwitterEmailCrash (computing)Open sourceDiagramDivisorRight angleComputer animation
02:19
Continuum hypothesisIntegrated development environmentAugmented realityVirtual realityReal numberType theorySystem programmingMixed realityPhysical systemReal numberDivisorElement (mathematics)Spectrum (functional analysis)DiagramPoint cloudObject (grammar)Mixed realityTerm (mathematics)Combinational logicAugmented realityVirtual realityStandard deviationPoint (geometry)Different (Kate Ryan album)Electronic mailing listVirtualizationCategory of being
03:40
Type theoryCategory of beingHacker (term)Term (mathematics)WordPhysical systemGame theoryComputer animation
04:30
Type theoryOpticsContrast (vision)Augmented realityRegular graphNeuroinformatikElectronic visual displayComputer animation
05:20
Video trackingInertialsystemMeasurementExecution unitDegree (graph theory)TrailArc (geometry)MereologyRotationRow (database)Term (mathematics)Maxima and minimaMaß <Mathematik>Degrees of freedom (physics and chemistry)Game controllerSpacetimePosition operatorFrequencyArtistic renderingMeasurementInertialsystemDisk read-and-write headError messageComputer animation
06:41
OpticsVideo trackingTrailTimestampOpticsSpacetimeRegular graphSlide ruleSquare numberOpen sourcePosition operatorView (database)Device driverWebcamPhysical systemDegrees of freedom (physics and chemistry)Machine visionAlgorithmLaserCombinational logicPrimitive (album)NeuroinformatikFrame problemLimit (category theory)Roundness (object)Set (mathematics)InternetworkingFrequencyCASE <Informatik>Computer hardwareTerm (mathematics)Similarity (geometry)BlogDistanceDemosceneSimultaneous localization and mappingComputer animation
08:49
Simultaneous localization and mappingTerm (mathematics)RoboticsPoint (geometry)NeuroinformatikMappingLocal ringMachine visionTrailAlgorithmDatabaseSpeech synthesisLevel (video gaming)Position operatorIntegrated development environmentFrame problemComputer animationLecture/Conference
09:26
Simultaneous localization and mappingPosition operatorRoboticsIntegrated development environmentLevel (video gaming)Demo (music)Medical imagingDifferent (Kate Ryan album)Type theoryGame controllerTouch typingDegrees of freedom (physics and chemistry)Simultaneous localization and mappingImmersion (album)Artistic renderingRepresentation (politics)outputTrailComputer hardwarePoint (geometry)Presentation of a groupRotationBitConnected spaceDisk read-and-write headComputer animation
11:29
Degrees of freedom (physics and chemistry)SpacetimeGame controllerImmersion (album)MereologyBranch (computer science)Term (mathematics)Forcing (mathematics)VotingSubject indexing
12:06
Video trackingClassical physicsMultiplication signTrailNeuroinformatikInteractive televisionDifferent (Kate Ryan album)Machine visionType theoryFeedbackRepresentation (politics)Machine learningSinc functionMechanism designTable (information)AngleArtificial neural networkVirtualizationUsabilityVirtual machineHand fanComputer animation
14:00
Video trackingGame controllerFeedbackVibrationSinc functionImmersion (album)Term (mathematics)Medical imagingOpen sourceProjective planeType theoryCrash (computing)Computer animation
14:47
Directed setAsynchronous Transfer ModeHead-mounted displayImage resolutionBit rateDigital rights managementExtension (kinesiology)WindowData managementElectronic visual displayLimit (category theory)Asynchronous Transfer ModeBit rateRun time (program lifecycle phase)Category of beingCartesian coordinate systemMultiplication signOpen sourceComputer animation
16:01
Directed setAsynchronous Transfer ModeHead-mounted displayImage resolutionBit rateDigital rights managementExtension (kinesiology)Demo (music)ImplementationBasis <Mathematik>Ext functorBasis <Mathematik>IterationOpen sourceAsynchronous Transfer ModeElectronic visual displayFrame problemVolumenvisualisierungCartesian coordinate systemExtension (kinesiology)ImplementationRun time (program lifecycle phase)DemosceneDemo (music)Front and back endsBit rateMoment (mathematics)Multiplication signGame theoryGraph coloringMathematicsEndliche Modelltheorie2 (number)Computer animation
18:00
Demo (music)Asynchronous Transfer ModeImplementationBasis <Mathematik>Directed setExtension (kinesiology)Communications protocolBranch (computer science)Video trackingSimultaneous localization and mappingTrailOpen setCodeClient (computing)Open sourceTrailDirection (geometry)Communications protocolImplementationComputer hardwareCartesian coordinate systemElectronic visual displayProjective planeLatent heatSimultaneous localization and mappingHead-mounted displayBranch (computer science)Digital rights managementAsynchronous Transfer ModeExtension (kinesiology)ConsistencyRun time (program lifecycle phase)Stability theoryRoboticsFlow separationVideoconferencingTouchscreenMultiplication signStudent's t-testPoint (geometry)Level (video gaming)Position operatorComputer animation
21:25
Projective planeAlgorithmPoint (geometry)Visualization (computer graphics)Position operatorRouter (computing)Normal (geometry)Configuration spaceWorkstation <Musikinstrument>CodeComputer fileView (database)TrailPersonal identification numberSystem callOpen setDemosceneComputer animation
22:25
Device driverGame controllerBranch (computer science)Point (geometry)TrailAlgorithmThree-dimensional spacePosition operatorProjective planeDevice driverSpacetimeCodeOpen sourceMereologyRevision controlBranch (computer science)Game controllerFront and back endsTask (computing)MultiplicationTraffic reportingImplementationLipschitz-StetigkeitComputer animation
23:58
Simultaneous localization and mapping1 (number)Interior (topology)Open setCivil engineeringImplementationDevice driverAsynchronous Transfer ModeDirected setDepth of fieldVideo trackingProjective planeSimultaneous localization and mappingDevice driverTrailDifferent (Kate Ryan album)Latent heatFamilyRevision controlCartesian coordinate systemAbstractionElectronic visual displayTelecommunicationImplementationOpen sourceMultiplication1 (number)WebsiteGroup actionVideoconferencingRun time (program lifecycle phase)Multiplication signLibrary (computing)Standard deviationSubject indexingOpen setLibrary catalogForm (programming)RWE DeaEndliche ModelltheorieExtreme programmingHeat transferTerm (mathematics)Hypermedia
27:27
Video trackingPrototypeVideoconferencingOffice suitePosition operatorAlgorithmTrailArithmetic progressionMultiplication signDevice driverProjective planeTerm (mathematics)Open sourceRun time (program lifecycle phase)Computer animation
28:40
outputFlow separationBuffer solutionOpen sourceWindowLibrary (computing)Data managementKeyboard shortcutGastropod shellGame controllerFreewareComputer animation
29:49
Demo (music)VideoconferencingDemo (music)Video gameDigital photographyEscape characterClosed setComputer animationEngineering drawingDiagram
30:28
TrailGame controllerError messagePropagation of uncertaintyBit rateGame theoryPosition operatorCursor (computers)Computer animationEngineering drawingDiagram
31:08
Cursor (computers)Game controllerPosition operatorQuantum stateGame theoryPointer (computer programming)Endliche ModelltheorieElectronic visual displayRight anglePoint (geometry)Overlay-NetzSpacetimeWindowComputer animation
31:46
Multiplication signTask (computing)Position operatorRoundness (object)WindowPeg solitaireSoftware testingEndliche ModelltheorieGUI widgetBuffer solutionData managementPhysical systemINTEGRALDemo (music)Computer animationLecture/Conference
32:26
EmulationGraph coloringWindowCartesian coordinate systemInterface (computing)Web 2.0Game controllerComputer-assisted translationBitComputer animation
33:04
Demo (music)Keyboard shortcutGroup actionAnalogyType theoryGame controllerDigital signalCloud computingGame controllerEndliche ModelltheorieSpacetimeoutputKeyboard shortcutBranch (computer science)Operator (mathematics)Game theoryPower (physics)Group actionCodeFile formatTask (computing)Physical systemEvent horizonCartesian coordinate systemPresentation of a groupAnalogyComputer animation
35:22
WindowMappingOperator (mathematics)Subject indexingSpacetimeDifferent (Kate Ryan album)Menu (computing)Set (mathematics)Game controllerRight angleGroup action
36:07
Computer wormStack (abstract data type)SoftwareStack (abstract data type)Cartesian coordinate systemRun time (program lifecycle phase)DemosceneBuffer solutionOverlay-NetzWindowType theoryAbstractionVolumenvisualisierungLibrary (computing)Object (grammar)Electronic visual displayOpen setRight angleGoodness of fitTraffic reportingRoundness (object)Multiplication signComputer animationProgram flowchart
37:12
Stack (abstract data type)outputEvent horizonKeyboard shortcutBuffer solutionWindowVolumenvisualisierungGame controllerRun time (program lifecycle phase)Cartesian coordinate systemDemosceneGame theoryOverlay-NetzLimit (category theory)Mobile appLibrary (computing)Flow separationMachine visionArithmetic meanMultiplication signPosition operatorFront and back endsData managementPlug-in (computing)RobotGastropod shellProgram flowchart
38:41
outputEvent horizonKeyboard shortcutBuffer solutionExtension (kinesiology)Buffer solutionFood energyOperator (mathematics)WindowShared memoryExtension (kinesiology)Limit (category theory)outputData managementIntelComputer animation
39:44
RectangleDemosceneWindowOverlay-NetzExterior algebraImplementationData managementDemosceneMenu (computing)Open setMedical imagingVolumenvisualisierungComputer animation
40:22
Core dumpFingerprintRepository (publishing)Open sourceSoftware developerLine (geometry)CodeMathematicsOpen sourceRepository (publishing)FingerprintFront and back endsGoodness of fitLevel (video gaming)Computer animation
41:18
Keyboard shortcutScripting languageDiscrete element methodDisintegrationEvent horizonoutputGUI widgetBuffer solutionPointer (computer programming)DemosceneGUI widgetImplementationOcean currentVirtualizationMultiplication signCodeData managementLibrary (computing)Endliche ModelltheorieWindowGame controllerKeyboard shortcutScripting languageInterface (computing)Raw image formatSampling (statistics)CASE <Informatik>Open setMathematicsDifferent (Kate Ryan album)SpacetimeStructural loadArc (geometry)Computer animation
43:14
Computer fontProxy serverCommunications protocolGame theory1 (number)Electric currentData bufferGraph (mathematics)GeometryPhysicsGUI widgetCombinational logicTotal S.A.Point (geometry)Communications protocolCartesian coordinate systemGeometryBuffer solutionVolumenvisualisierungPosition operatorInclusion mapRootPhysicalismOrder (biology)Computer animation
44:25
Communications protocolGame theory1 (number)Electric currentData bufferGraph (mathematics)Proxy serverPhysicsGeometryBuffer solutionInterface (computing)PhysicalismWindowData managementWide area networkProjective planeCartesian coordinate systemCommunications protocolEndliche ModelltheorieClient (computing)Demo (music)SurfaceMereologyCuboidComputer animation
45:02
Online chatFreewareMereologyCartesian coordinate systemKeyboard shortcutCommunications protocolGame theoryWindowOpen sourceProjective planeData managementArithmetic meanPointer (computer programming)Game controllerInteractive televisionMathematicsTwitterPhysicalismSlide ruleVideoconferencingWikiMultiplication signRepresentation (politics)CubeSoftware developerOverlay-Netz1 (number)Real numberGoodness of fitOffice suiteCellular automatonView (database)Point (geometry)PlanningDirection (geometry)Open setComputer programmingIterationComputer animation
49:26
Point cloudOpen sourceComputer animation
Transcript: English(auto-generated)
00:05
Hello. Welcome to the talk from Lubos about the year of the Linux virtual desktop. Please, a round of applause for Lubos.
00:22
Yeah, hi. I'm Lubos Saunetski. I'm at Collabora. And today I'm going to present about XR on Linux in general in open source and the Linux desktop in VR, so mostly related to my work.
00:45
And thank you all for attending, for your interest in my work, and also thank you for Fostum for having me and considering me for the main track. This is my third Fostum, but it's the first time when I speak, so I hope it won't be the last time.
01:02
Usually I don't put up the date on my slides, but today's date is so nice. It's a palindrome, so you can read it both ways. And it's the same in American and in normal notation. So this is me with a funny headset. It's an AR cardboard phone holder headset.
01:23
If you have any complaints or questions about the talk, please contact me on email or Twitter. So I will start with a brief crash course about what XR is, what kind of terminology we have there.
01:41
Maybe I should start with just asking you how many of you have ever tried AR or VR before. That's plenty. That's nearly all of you. How many of you did that on Linux? Okay, that's way less. And how many of that people that did that on Linux did that on a completely open source stack?
02:06
That's even fewer. Maybe nearly none. And this is where we are working on right now currently. So I hope that this year 2020 will improve the situation. There are some factors that will help with that.
02:22
But let's first start with this nice diagram. It's by a guy called Milgram from 94. So VR is actually quite old already. And on this spectrum we have the real world, the world we are in right now on the left side, and a completely virtual environment on the right side.
02:43
And in between there are some steps. So augmented reality is, for example, if you have virtual elements in the real world, like annotations for real objects, and then we can go to the completely virtual world. And on the way we found the augmented virtual reality, this is when you have, for example,
03:04
camera feeds or 3D point clouds of real objects in the virtual world. And this spectrum was called mixed reality spectrum. And the terms, I have listed the terms here again.
03:22
And a more prominent term nowadays is maybe XR, which is called X reality or cross reality. This is also what's the term that is used in the chrono standard, OpenXR for example. And this is the term I'm going to go with, because it just summarizes all of these combinations of realities, of inter-reality systems.
03:47
And in terms of consumer available headsets, I would just point out three categories we have there. For example, we have the simple phone holder headsets, which run the phone's operating system,
04:04
which render on the phone, and you have simple lenses in them. These are the most accessible. These are what people mostly use. Most people have contact with them. Of course, you can get it fancier as a piece of paper, but it's as simple as it can get.
04:22
And of course, we have the PC tethered headsets. That's something for the PC gamer enthusiasts. I guess they're also more hacker friendly, since you can use your desktop, your regular Linux desktop, and use the device as ever you like, because you're rude.
04:41
And on the right side, I will point out the standalone headsets, which are similar to the phone thing, but like built from the ground up. So you have an embedded computer in there, maybe in a belt, or maybe also integrated in the headset. And the lenses are better adjusted for the display, in contrast to these phone holders, which are more generic.
05:08
And this is, for example, an augmented reality headset with optical see-through, so you can actually see the real world. And if you want to do see-through on a device like that, you usually have a camera where you do video see-through.
05:25
In terms of tracking, so tracking is the most interesting part, I guess, when it comes to XR. The rendering is more straightforward than that. For tracking, one prominent device that also is in your phones is an inertial measurement unit, so-called IMU.
05:47
It's pretty cheap and pretty small, so this is an example how this looks. It has multiple sensors, so it has, for example, a gyro and accelerometer. It also has a magnetoscope, so a compass, but usually that's not used in terms of error.
06:07
And it's a very high frequency sensor, so compared to the optical sensors I will show you on the next slide, you get a very high frequency signal. And, yeah, this is how VR, what the minimal requirement is for VR,
06:26
but you only get three degrees of freedom out of that by itself, so that means only your head's rotation will be recognized by the tracking system, or if it's a controller, it will be only the rotation of the controller and not the position in space.
06:44
If you want a position in space, you need to have optical tracking. For example, this is the work of Philip Zabel. He's implementing an open source driver for the Oculus CV1, and this is exactly this headset seen in an infrared view.
07:04
So the camera is a USB camera and a regular USB camera with either an infrared filter or an infrared sensor, in best case, and this is what the camera sees. The regular person or the human eye wouldn't see the slide because it's out of our visible spectrum.
07:22
I'm not sure if you can see it, but there are like squares around these blobs, these so-called blobs with primitive computer vision algorithms. These blobs can be detected, and with these blobs the position can be calculated off the headset in space.
07:40
In combination with the IMU you saw on this slide before, there needs to be something done called sensor fusion. So we have the data from one sensor from the camera, which runs at 60 Hz or something, at regular camera frame rates or frequencies, and the IMU runs at much faster frequencies,
08:04
I guess about 10 kHz, and this needs to be fused. So this is one style of external optical tracking. The other style is the other way around, where the camera emits the laser and the sensor is on the headset.
08:24
So in this case, for example, this is the lighthouse tracking system. It has two rotors where two lasers rotate, and when they hit the sensor on the headset, you get a timestamp and then you can calculate something similar like here.
08:43
And from these techniques you get six degrees of freedom. That means you can also recognize if the user moves up or down or in space. Another way of tracking, which is easier in terms of hardware, is SLAM, that's from robotics.
09:04
It's called Simultaneous Localization and Mapping, but basically in XR we call that inside-out tracking. That means that you have a camera in the headset and we run some computer vision algorithms to get the feature points, and then the feature points are stored in a database
09:26
and you can see if you saw this feature in the last frame. And from this you can also calculate the position. And in robotics it's also used for calculating a map of the real environment.
09:45
So the SLAM is a very popular method of tracking because it's easy to implement in hardware, but it's a bit tricky to do that in software, especially if you want things like low latency. The next point is input in VR, because not only the rendering is interesting and gives us immersion,
10:09
like stereo rendering for one image for each eye, it's immersive definitely and if you move your head. But one thing that gives us even more immersion, so more connection to reality, is the input in VR.
10:25
For example, when I first did a VR demo back in 2013 without controllers, because it wasn't a thing back then, people asked me, oh, where are my hands in VR? I would like to see a representation of my body. And with tracked controllers this is now more possible and the experience can be more immersive.
10:47
And I have two different types of controllers here. The one on the left is a simple controller with just an IMU, so you just can get the rotation from it, and it has a touchpad and a button. It doesn't even have a trigger.
11:05
But for simple things like pointing on something and clicking on it, it's enough. But on the right side we have a more complex controller, which also has things like finger tracking. So here on the left side it has a proximity sensor and it detects how far your fingers are away from the controller,
11:28
which gives you already the possibility to have a virtual representation of your hand. It also has six degrees of freedom tracking, so you can actually position your hands in space correctly,
11:43
which gives us more immersion. So in terms of what devices these are, this is the Daydream controller. We have a branch in Monado that supports that from Peak Black.
12:00
But I will come to details what Monado is and where you can get that later. And this is the Valve Index controller, a pretty nice controller. And the best thing to use in VR for interaction is something we have with us all day.
12:20
It's our body and hands since with our real hands we can experience the things even more realistic. It's like when you see your hands in VR that actually look like your hands, you recognize this from the real world because you maybe notice that you look at your hands quite a lot.
12:45
Like when I pick this bottle I'm actually looking at my hands and I know how they look. So if you have like a weird virtual representation of different hands, it's not as immersive as if you have your real hands. And hand tracking can be done by pure computer vision.
13:01
This is a stereo camera which is also infrared. It emits infrared light and it's a wide-angle stereo camera with some computer vision and machine learning even. There can be a virtual representation of the hands calculated from this image.
13:23
On the right side I have a more classical approach at hand tracking which is mechanical. It has the disadvantage of not being very user-friendly so you need some time to put this on. It's not like this where you don't need to do anything, you just have your hands.
13:44
But it has the advantage of haptic feedback which is something not there with the other type of hand tracking. And in VR in general haptic feedback is not something we have like in the real world so I cannot really touch the table and lean on it in VR.
14:02
Something I can do is maybe vibrate, let the controller vibrate or maybe do acoustic feedback that replaces the haptic feedback because the sensorics in the brain are very tolerant so you can do acoustic feedback and the human mind will find it okay in terms of immersion.
14:24
So this was the rough crash course about what we have in XR, what the challenges are, what the types of devices are. And now I wanted to point out some projects that implemented that in open source.
14:50
So one important thing that needed to be done in the Linux graphics stack is something called direct mode. It's the possibility to lease the display of the headset
15:06
so it's not used by the window manager as a desktop display and the application or the XR runtime can render to it directly. It's based on work by Keith Packard and he introduced the non-desktop property on displays.
15:30
You can see it in XR and R maybe and you are wondering what this is. And it has the advantage of not only that windows don't show up, like desktop windows don't show up on your display by mistake,
15:45
it also has the advantage that you can render with a native refresh rate on the HMD. For example, HMDs have usually a refresh rate of 90 or like the more modern one, 144 and like all the desktop displays have just a refresh rate of 60.
16:04
And in the old days of open source VR we were unlucky to run the headset at the same refresh rate as the desktop. So this was quite bad and this was resolved, I guess early 2019 this landed in the stack
16:24
so it needed a couple of changes, so you need a fairly recent mesa for that and fairly recent XR and R and stuff and Vulkan as well. So the Vulkan extension is called acquired XLIP display and as soon as this was like available I implemented it in XR gears.
16:46
This is a demo application I wrote that renders a stereo scene in Vulkan and it is the gears to know them and it uses this extension as a reference implementation.
17:05
I also had other backends like extended mode, so if you don't render directly to the display it's called extended mode. And I had extended modes in Wayland and XCB and also a KMS backend I guess for Intel.
17:24
The work I did here later became the basis for the compositor in Monado. So Monado is our open source runtime we develop at Collabora, our open source OpenXR runtime and one aspect of the runtime is to provide all of the stuff
17:43
so the application doesn't need to deal with it. So the runtime has this compositor that opens, it leases the display and you only have the standard API to submit frames to that. And later I made a second iteration of this application
18:02
which is just an OpenXR client which makes the code reduces it by one half at least. And on Wayland the situation was different so this needed to be implemented on Wayland as well.
18:25
And if you want to know details about direct mode on Wayland it was mostly done by Drew Devault and protocol was specified by NXP, it's called DRM lease on stable v1 and the Vulkan extension is also called similarly as for X.
18:44
This also supports XWayland clients that means you can run an X application with that as well and it will utilize the direct mode. This is a screenshot from Drew, he did that through the lens so you can actually see that it runs on the display.
19:04
And it's not like all of this is not upstream yet I guess there is a merge request for the Vulkan specification. This protocol needs to be implemented in the Wayland compositor so you need to have a compositor that implements that and there is a branch for Monado for the runtime
19:22
so it can actually utilize that. So now we have the HMD working, like the display working and we need to get some tracking data. I wanted to point out some notable open source tracking projects
19:43
most prominently maybe OpenHMD. Funny thing is that OpenHD has a BoF going on right now so they are not attending at this talk I can talk whatever I want about them Hopefully I won't see the video No, but OpenHMD is a community of enthusiasts
20:00
and they have their methods and tools to get support for hardware pretty quickly so they analyze the hit protocol the display is having the HMD is having over USB with the PC and try to implement that. And mostly what is there in OpenHMD is treat-off tracking
20:24
so they have... it's rather quickly to get a new headset supported with treat-off tracking so just the IMU but it's more complicated to get positional tracking done and OpenHMD is currently implementing positional tracking
20:41
for the Oculus CV1 This is work done by Philip Zabel and Jan Schmid They also did a couple of talks about this in the past Maybe other projects I should point out are here So there are many projects that actually provide low-level access to the device
21:04
but they don't provide a consistent API for applications and they're rather like experiments Also one SLAM I want to point out is MapLab It's a pretty decent SLAM SLAM is usually used by robotics and used in research a lot
21:23
but there are several open source implementations What I was doing back in the day This is from 2017 We worked on a project called Vive Libre and here you can see the Lighthouse tracking
21:43
we implemented So this code is based on OpenHMD and the Lighthouse Redux documentation and this is just a MapLab visualization of what we have here So here are these points It's the headset basically seen from the base station
22:03
the one with the routers you've seen before and this is the configuration file from the headset You can read out as JSON These are the positions and the normals of the sensors on the headset and with the data from the sensors we could reconstruct this so-called station view
22:24
and with a simple algorithm from OpenCV it's called PNP, point and point algorithm You can give this algorithm a bunch of 2D points and a bunch of 3D points and then it calculates the position in 3D space according to the camera
22:42
and so this was when we prototyped the Lighthouse tracking and the project or the code was picked up by a project called LibSurvive and it is a free and open source Lighthouse driver They added many things to our work
23:03
like proper filtering So filtering is the complicated part of getting multiple sensor data so sensor fusion and filtering are kind of synonymous and they also added support for controllers and also support for newer headsets
23:22
and they have multiple so-called poses just multiple backends that are competing at the same task to figure out which approach is the best The project was done by a guy called Cian Lor This is him running the code on the Raspberry Pi
23:41
I guess he's also doing the rendering on the Raspberry Pi which is pretty nice and this is not all Raspberry Pi I guess this is a version 2 or something And we have a branch My colleague Christoph implemented LibSurvive in Monado So it's actually now usable with the OpenXR API
24:04
So you don't need to read this Just to show how many open source SLAM implementations there are There are quite a lot They have different versions and different quality of projects Most of them are from academia
24:21
and there is a site openslam.org that catalyzes all of them But for Monado we will choose one for you in the near future So you don't need to choose that But there is a lot of going on in open source SLAM I mentioned this a couple of times now
24:45
OpenXR is a standard API from Kronos like OpenGL and Vulkan And it tries to achieve a problem the industry had before OpenXR Before OpenXR we had only vendor specific APIs
25:01
So most vendors that did a headset also did their API That means that applications were not portable Mostly the abstraction was done by the engines like Unity and Unreal So most of the projects used, for example, Unity so they could run on multiple vendors
25:21
But OpenXR tries to standardize that And we hope that it will get rapid adoption It was released last year, so early 2019 or something like that So it's still new But most of the big vendors are in the specification
25:44
or in the group that specifies that And we at Collabra decided to implement this API as well And we did that in Monado our open source runtime
26:00
So Monado implements OpenXR and provides the device drivers from, for example, OpenHMD or libsurvive But it also has its own device drivers that we just write for Monado in our internal driver library For example, we have device drivers for the OSVR HDK2
26:26
or we are currently working on a PSVR driver which also will have positional tracking So my colleague Pete Black is working on that And I also recently wrote a native driver
26:43
for the Vive and Index family of headsets that's currently quite simple but it will evolve in the future So we, as I mentioned already, we have a Vulkan compositor that opens the display for the application
27:00
and communicates with that And we're currently working on 6DoF tracking And, as I mentioned, we are also looking into providing SLAM for that So Monado manages the devices the camera devices, for example from not only the cameras that track the headset
27:22
but also cameras that are on the headset that could be used for video see-through, for example And this is a video from my colleague Pete Black He gave it to me today I think he never released that It's an example of his PSVR tracking
27:43
I'm not sure if you know how the headset looks but here you see the emitters on the headset and the positions that are calculated by his algorithm So this is work in progress and soon you will have that exposed through the OpenXR API
28:05
So this was an overview of what's going on in terms of drivers and tracking and runtimes And now, since we have that now, kind of
28:20
we can build on top and what do we want to do with that Now we have VR in Linux or in open source And one thing that can be done with that is the project I've been working on for the recent time and it was released, I guess mid-2019
28:42
So it's XR Desktop And in XR Desktop we have made stack of libraries that interface with existing window managers or compositors like this, this is for example GNOME Shell
29:04
We get the window buffers from GNOME Shell and can display them in VR And what we also do is to get the input from the controllers and synthesize this so the 2D desktop receives mouse and keyboard strokes
29:26
3D desktops are not new, like there were several open source 3D desktops or VR desktops But what is new in our approach is that we try to interface with the existing window manager
29:41
so that you don't run the 3D desktop separately separated from your regular desktop but you can just mirror the existing desktop And this is something that was sponsored by Valve So thank you for that Without them, XR Desktop wouldn't be what it is right now
30:04
And I have a demo, not a live demo, but I have a video Since doing live demos with VR is kind of crazy I guess But I have colleagues that do that
30:21
So this is me using Inkscape in VR So what you can see here is that I'm dragging the window around and I'm moving it close to me So if you move things closer then you have more precision You can of course use it from far away
30:42
but there will be an accumulation of error in the tracking, of handshaking This is also something you have to think about because you are holding the controller in your free hand and it's not on the table like with the mouse So you have handshaking And I'm showing my insane drawing skills in Inkscape here
31:09
As you can see we have the cursor that is shown on the desktop as well in VR and it is shown at the position of the end of your ray, so called ray from the controller
31:24
And this is just one game from Gnome Games that I tried to solve As you can see we also, when you click on the end of the pointer there is an animation emitted And this was a model dialogue right now
31:41
So if the model dialogue is opened we overlay it in 3D space over the window as well which has had to be implemented It's not like appearing in the zero position of the world And solitaire is actually a nice thing to do in VR
32:02
Every time I do a longer test I play a round of solitaire I guess So this is Krita, also has a model dialogue So the actual widgets rendered as they are done usually And we get the buffer from the window manager
32:23
And this demo is showing Gnome but we have also integration for KDE And I'm using Krita to draw in VR Of course there could be many improvements For example one feature I heard many people would like to have
32:42
also my colleague Christoph is that you put like other windows on the right hand and you could just use the color picker on your hands and to interface better with the 2D application So let's skip a bit forward
33:01
I'm browsing the web with cats So on the controller you have a touchpad and you can just scroll feeds as you are used to And this is also another example of a model dialogue So it's quite usable if you get used to it
33:35
So another interesting concept in XR is the concept of actions
33:43
In traditional games or input systems like SDL you are listening for example for the space bar or looking if you got an event from the space bar press or release or mouse clicks And with an action system this is kind of decoupled
34:02
So now we don't check if the space bar is pressed but we check if the user wants to jump for example This is necessary since XR controllers are very heterogeneous and the hack the PC gaming industry did and just supposed everyone has an Xbox controller
34:24
doesn't work anymore So there is something called actions and bindings and the actions are defined by the application I want to jump and this is a boolean operation or I want to move forward and this is an analog operation for example
34:41
and the bindings need to be created by the runtime or by the user that has the device That means I have this controller here this presentation controller and if I press the right button the thing should jump And this is available in the OpenVR API
35:04
which is the API SteamVR uses and as well in the OpenXR API Usually like in OpenVR this is specified by JSON In OpenXR it's specified in code but on our OpenXR branch for extra desktop
35:22
we also have a JSON format that's quite similar to this one So this is how the mappings look for extra desktop in particular This is the index controller We have two different sets of actions One set is for interacting in 3D with the windows
35:42
and the other set is for doing the 2D desktop operations like right click, left click and scroll and the 3D operations are pushing and pulling the window in Z space or on the Z axis and opening menu and stuff like that
36:01
So this is also translatable to the other controllers So let's look at some software in our stack On the very bottom we wrote Like our stack is written in glib
36:21
and we introduced a couple of glib libraries For example Golcan is our Vulkan abstraction library where we do some things required to display windows in 3D and render objects We have GXR which is the library
36:41
that abstracts both OpenXR and OpenVR APIs So you can write an application in GXR that actually runs on both APIs And in XR desktop which is also a library we provide two types of applications The one is an overlay application
37:01
This means that the scene is rendered by the VR runtime and we just supply the windows and we have also a scene application which means where we render the full scene where we render the stereo buffers and submit the final stereo buffer rendering to the runtime The overlay app has the advantage
37:21
that you can see the windows drawn over a VR application that you're running It means you can play a game and see desktop windows and the other, the scene app has the advantage that we have full control of the renderer and can display as many windows as we want
37:41
The other one has limitations libinputsynth is the library where we do button presses and mouse clicks and mouse position manipulation We have several backends for that For example XDO on X11 and there is also stuff for Wayland in there
38:00
And this is how we interface basically with the window manager On kwin for KDE there is a plugin that can be loaded which is quite convenient since we don't need to fork kwin On gnome-shell the situation is different We need to fork gnome-shell to interface with it
38:22
But this will hopefully change in the near future when I do my upstream work So again, libinputsynth was required so we can actually also do clicks on the desktop
38:41
that runs at the same time as we do it in VR which wouldn't be required if we had a stand-alone VR compositor This is just required because we want to run on both Yes, and I talked about it already
39:01
So how do we share the window buffers? We do zero copy operation so we don't download the windows from the GPU All the window buffers stay on the GPU We use a GL Vulkan interop because sadly the compositors are still using GL
39:20
which is not a big problem since there are extensions available to share memory between them So this is how the extensions are called in Vulkan and GL accordingly Unfortunately on Intel this extension is not implemented as far as I know so this is a limitation
39:41
On Intel the extension is not implemented in GL That means if we had a Vulkan window manager it would be fine And the overlays are currently only implemented in the OpenVR implementation of XR desktop
40:01
since the OpenXR alternative for that is called XR composition layer is currently not implemented in Monado so we are working on that as well So this is an example of the scene renderer with my example image
40:21
the hawk from Wikipedia We can render many hawks Our upcoming release of XR desktop is coming soon So shortly after FOSTERM we will release 0.14 It's our biggest release yet with the most lines of code changed and most commits
40:41
Like we have about 364 We are two developers We changed a bunch of APIs so it can run OpenXR and OpenVR backends with the same code So look out for that It's already on Git master If you wonder where to get XR desktop
41:01
If you are arch user you are lucky We have very good arch user repository packages We have Ubuntu PPA which is maintained but maybe not updated currently but we are working on that as well and of course you can build it from source We have a Vicky article on that
41:24
So let's take a quick look in our roadmap What are we trying to do in the near future? We would like to have a virtual keyboard Currently on OpenVR we are using the one from SteamVR On OpenXR we currently don't have one So it would be nice to have the same one on both APIs
41:42
We want to implement GLTF loading So we also have the controller models on the OpenXR implementation and can also maybe load other cool models like a scene where you spend your time and when you are using your desktop Scripting is also something we are looking out to and for the maybe not so near future
42:04
We wanted to do a 3D widget toolkit I would call that G3K because we have a lot of code that is related to that but it's just not nicely in one library and it doesn't have a nice API And with that we could do something that's called maybe
42:21
XR Desktop Shell where you have not only windows floating around but maybe a full desktop experience with a clock and widgets and workspaces So this is the future Also what's maybe interesting is how to interface with 2D toolkits
42:41
So we can also render from the 2D toolkit directly into VR without using the window manager which is nicer since we can do stuff like high DPI and maybe nicer fonts For that we need zero copy access to the toolkit
43:00
GTK3 had something called GTK Offscreen Window and GSK is different We need to access GTK4 It has now a scene graph, it's called GSK so they change a lot and I have an example for GTK3 for that
43:21
Native 3D UI, so what do you need a 3D widget toolkit for? It's a nice thing since you can have actual 3D widgets This is maybe not 3D enough This is like a combination between 2D and 3D but you can also have like total 3D UI like you would work with the reality
43:42
and this is I guess the way to move forward to have native 3D applications Also font rendering I want to point out Mozilla Pathfinder which is font rendering for XR and it's Apache licensed And one more interesting thing
44:01
is what about 3D applications, native 3D applications? It would be nice to have a protocol for that So the renderer for the 3D application could render for example a stereo buffer and receive the position from the compositor or we could also supply native geometry to the compositor
44:21
which renders the 3D application This would have the advantage that we could do transparencies, occlusions and physics if we use a real model for that Drew Devault worked on that He has in his project called WXRC
44:41
It's a 3D window manager or VR window manager and it can take 3D applications as clients So he designs a WAN protocol for that It's pretty neat, check it out This is a demo where he opens 2D surfaces first
45:05
but the most interesting part for me was when he opens a 3D application And this is also something I would like to see in XR desktop and since he specified a protocol
45:21
it's pretty neat to implement that And this is the 3D cube Unfortunately, sometimes LibreOffice doesn't want to skip this slide if I have a video No, it did OK, if you want to get involved
45:41
We are on Freenode or we have a Discord and look at our wiki on Twitter and GitLab There is also a FOSSXR conference It will be held again in Amsterdam That's the second iteration So if you are interested, it's around Blender conference
46:01
So we are looking forward to see you there And I guess now we can come to questions So, any questions?
46:25
Do you see a future for the desktop with VR but with mouse and keyboard interaction like we just saw here at the end? So the question was if I see a future for mouse and keyboard interaction in VR So definitely, as a programmer, I enjoy to have a physical keyboard
46:44
And for this, we would require either virtual representation of the keyboard in VR or you just have augmented reality where you just use your real keyboard and can see that So definitely, there are also projects
47:04
For example, the window manager I showed in the end I guess it has only keyboard and mouse support In XR Desktop, it's kind of complicated since we moved the mouse pointer around This could get quite hacky to decouple that from the real mouse So I wonder how to solve that
47:21
But with standalone VR compositors, it's quite possible I was wondering, is it compatible with other VR applications?
47:44
Like, what if you want to start a VR game inside it? Does that require any kind of special support for it to work? Do you mean XR Desktop or OpenXR? XR Desktop, yes So XR Desktop can run on top of a VR application
48:03
if you have it in overlay mode, that's possible So this is also one feature we intended the design to be that we can run XR Desktop, you can show existing desktop windows over VR applications So this was one goal
48:20
Currently on OpenXR, it's not supported because this would require some changes in Monado which are going to happen sometime So, who are you targeting with OpenXR and Monado?
48:45
With OpenXR? Yes I hope that OpenXR is... We're targeting OpenXR for all XR application developers So they write applications for the same API and not only application developers, but engines in particular
49:00
There are many engines already supporting OpenXR like Unreal, Unity, Godot engine for example There was a talk yesterday by my colleague Christoph He implemented support in the open source Godot engine for OpenXR Any more questions?
49:24
Thank you for your time