Holographic Programming: Exploring the HoloLens
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 96 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/51722 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
NDC Oslo 201636 / 96
2
7
8
9
12
14
19
20
26
28
31
33
38
40
43
45
48
50
51
61
63
65
76
79
80
83
87
88
90
93
94
96
00:00
Computer programmingConvex hullSocial engineering (security)BitEqualiser (mathematics)Water vaporComputing platformWindowSpeech synthesisQuicksortMultiplication signMoment (mathematics)Quantum statePrice indexCategory of beingInternet service providerMessage passingTape driveMixed realitySoftware testingGame theoryRight angleComputer animation
02:30
Self-organizationSoftware developerGame theoryGoodness of fitState of matterLink (knot theory)CollaborationismOptical disc driveLimit (category theory)TouchscreenPoint (geometry)TwitterData managementSoftwareVideoconferencingContext awarenessMultiplication signBitQuicksortComputer programmingOcean currentCountingOpen setArmGraphics softwareComputer animation
05:15
PrototypeVirtual realityDisintegrationAugmented realityMixed realitySurfaceDigital signalQuicksortInteractive televisionBitVirtual realityDigitizingVideo gameSurfaceState of matterMixed realitySmartphoneNeuroinformatikWindowGoodness of fitComplete metric spaceNatural numberPeripheralSphereObject (grammar)Augmented realityLevel (video gaming)Web-DesignerRight angleGodCASE <Informatik>Multiplication signSquare numberTerm (mathematics)Mobile appTouchscreenEscape characterDemo (music)Chemical equationTouch typingSoftware developerCodecBuildingService (economics)LogicArc (geometry)Computer animation
09:00
Real numberVideo gameSupersonic speedVideoconferencingMathematicsDifferent (Kate Ryan album)Virtual realityAugmented realityComputer animationLecture/Conference
09:48
PlanningDifferent (Kate Ryan album)Mobile appAugmented realitySemiconductor memoryVirtual realitySoftware testingSystem callDemo (music)CyberspaceGame theoryBoss CorporationDisk read-and-write headComputer hardwareBitLattice (order)Level (video gaming)
11:11
Data storage deviceComputer hardwareAbstractionCuboidLaptopWindowRight angleMobile appUniverse (mathematics)BitCoprocessorData storage devicePolygonQuicksortField (computer science)SoftwareCyberspaceGoodness of fitGodSoftware developerDisk read-and-write headSemiconductor memoryPattern recognitionView (database)Game theoryVideoconferencingSpeech synthesisProcess (computing)TwitterFitness functionNeuroinformatikComputing platformFacebookPrototype2 (number)Demo (music)Beat (acoustics)Regular graphTouch typingCollineationMultiplication signDifferent (Kate Ryan album)Power (physics)Projective planeComputer clusterMessage passingWordForceComputerVotingMultilaterationComputer animation
16:51
Computer clusterComputer hardwareWhiteboardSelf-organizationConfiguration spaceComputer animation
18:35
Level (video gaming)Game theoryRight angleGame controllerVideoconferencingSurfaceDemo (music)Perspective (visual)Covering spaceService (economics)Physical lawTrailTable (information)Computer animationLecture/Conference
19:44
Different (Kate Ryan album)TouchscreenGame theoryVideo gameBuildingBlock (periodic table)Theory of relativityCursor (computers)Element (mathematics)Computer animationMeeting/Interview
20:36
Element (mathematics)Cursor (computers)Quantum stateComputer programmingVolumen-RenderingObject (grammar)Pointer (computer programming)Casting (performing arts)Mountain passComputer animation
21:24
OvalScripting languagePolygon meshOrientation (vector space)Group action1 (number)Mobile appDemosceneMeasurementGoodness of fitObject (grammar)Matching (graph theory)Multiplication signScripting languageCursor (computers)CASE <Informatik>Ring (mathematics)Price indexInformationRight anglePay televisionDigital photographyRoboticsProcess (computing)Game controllerExecution unitComputer animation
22:54
OvalOrientation (vector space)Polygon meshUniform resource locatorCursor (computers)Electronic visual displaySinguläres IntegralRotationSurfacePhysicsVolumenvisualisierungForm (programming)Ring (mathematics)Mobile appPhysicalismDisk read-and-write headInformationVolumenvisualisierungBitPolygon meshCursor (computers)Position operatorMultiplication signRotationException handlingObject (grammar)Frame problemVolumen-RenderingGoodness of fitTransformation (genetics)Point (geometry)Key (cryptography)Zoom lensCodeScripting languageEmulatorArrow of time2 (number)Video gameSurfaceBoolean algebraQuicksortThree-dimensional spaceMenu (computing)CASE <Informatik>MeasurementCoordinate systemGame theoryBit rateMachine visionCyberspaceBuildingHand fanRight angleCausality1 (number)Computer animation
26:46
Demo (music)BitInteractive televisionQuicksortElement (mathematics)Disk read-and-write headFrame problemObject (grammar)Multiplication signMenu (computing)Mechanism designThomas BayesRoboticsRing (mathematics)3 (number)Group actionComputer virusField (computer science)CyberspaceView (database)Computer animationEngineering drawing
27:58
1 (number)Instance (computer science)Object (grammar)OvalInformation managementFinite-state machineEvent horizonCountingMessage passingView (database)Orientation (vector space)Group actionPhysicsDialectView (database)Menu (computing)Field (computer science)Medical imagingGame theoryFocus (optics)Multiplication signData managementCasting (performing arts)FreewareBit rateFrame problemGroup actionEvent horizonMathematical optimizationRight angleVolumen-RenderingBitLink (knot theory)TwitterSelectivity (electronic)Computer animation
29:52
OvalPhysicsSphereLocal ringFatou-MengeContinuous functionEmulatorKey (cryptography)Rigid bodySelectivity (electronic)Object (grammar)PhysicalismThree-dimensional spaceMathematicsPosition operatorArithmetic meanCyberspaceMassSoftware developerComputer animation
30:54
PhysicsContinuous functionMenu (computing)View (database)Disk read-and-write headEmulatorWebsiteRight angleVideo gameHypermediaInformationMereologyMultilaterationSquare numberPlanningMultiplication signPeripheralSurfaceView (database)TouchscreenBitKey (cryptography)Computer animation
32:53
PhysicsOvalSphereLocal ringObject (grammar)Euclidean vectorHost Identity ProtocolView (database)View (database)Computer hardwareTheoryTerm (mathematics)Process (computing)VideoconferencingQuicksortPrototypeField (computer science)PlanningPower (physics)Revision controlExecution unitComputer animation
34:38
Group actionSelectivity (electronic)Library catalogFunction (mathematics)INTEGRALInternetworkingWordMereologyTheorySpeech synthesisBitRoundness (object)WindowRight angleRemote procedure callConnected spaceBoss CorporationMobile appSource codeLecture/Conference
37:00
Demo (music)OvalSpherePhysicsPhysical systemFocus (optics)Object (grammar)Instance (computer science)Indian Remote SensingSystem callMIDIView (database)outputWindowGoodness of fitRevision controlMachine visionAndroid (robot)Finite-state machineGreatest elementDrop (liquid)Functional (mathematics)Link (knot theory)Different (Kate Ryan album)Event horizonMobile appMereologySpeech synthesisString (computer science)Software developerEmulatorPoint (geometry)Right angleSphereCodeBroadcasting (networking)Key (cryptography)Electric generatorMessage passingForcing (mathematics)WordBlock (periodic table)Uniformer RaumInheritance (object-oriented programming)BuildingComputer animation
41:07
NoiseHeat transferFunction (mathematics)Block (periodic table)Speech synthesisBuildingDirection (geometry)SimulationBitHeat transferGoodness of fitDemo (music)Disk read-and-write headÜbertragungsfunktionMassComputer animation
42:39
Euclidean vectorDefault (computer science)Source codeCompilation albumExecution unitMaxima and minimaMobile app1 (number)SphereRight angleBuildingBitVisualization (computer graphics)MereologyProjective planeACIDUsabilityScripting languageTerm (mathematics)Moving averageOpen setComputer animation
43:52
OvalEuclidean vectorDefault (computer science)Object (grammar)Game theorySource codeVolumeStructural loadSphereHill differential equationCollisionMoving averageEmulatorObject (grammar)CodeNoise (electronics)NeuroinformatikSource codeDemo (music)Right angleDisk read-and-write headVelocityQuicksortCollisionMoving averageMathematicsComputer sciencePhysical lawProjective planeComputer animation
46:00
CodeComputing platformMobile appWindowBitTouchscreenWeb 2.0Revision controlUniverse (mathematics)BuildingNumberEngineering physicsCodeRight angleData storage deviceEmailEndliche ModelltheorieComputer animation
47:58
Complete metric spaceWaveBuildingSoftwareMobile appWaveTheory1 (number)CyberspaceEndliche ModelltheorieLimit (category theory)Table (information)Right angleInteractive televisionReal numberMassEmulatorInheritance (object-oriented programming)Software developerQuicksortBitFreewareType theoryVisualization (computer graphics)Mixed realitySphereVirtual realityAreaTrailAdditionProcess (computing)Multiplication signControl flowMeeting/InterviewComputer animationLecture/Conference
51:50
AreaSpherePolygonWave packetProcess (computing)Bit
52:55
Service (economics)Limit (category theory)PhysicalismMultiplication signTable (information)CyberspaceMonster groupInternetworkingVideoconferencingConnected spaceMobile appAngleEndliche ModelltheorieObject (grammar)SurfaceReal-time operating systemComputer clusterCasting (performing arts)BuildingElement (mathematics)1 (number)BitImplementationSlide ruleData miningLogicMoving averageType theoryUniform boundedness principleSpeech synthesisSphereVolumen-RenderingQuicksortComputer animation
57:14
Multiplication signUniform boundedness principleBitSlide rule2 (number)Computer animation
Transcript: English(auto-generated)
00:04
Good morning How cool was that press play to on tape? What was it called? Press play on tape last night. Holy crap Make me feel really old because I knew all the games Thanks for coming. I Just want to sort of start with a little bit of disclaimer. Did anyone yesterday go to Renee's talk?
00:24
Just to show hands, please. It's a few. All right, so there was a HoloLens talk yesterday and And Renee actually has a HoloLens and I don't Know it's good for you because I'm guessing you don't have a HoloLens either. Does anyone have a HoloLens? Jimmy hands down
00:43
No, and that's the that's the whole idea right there. There are really none of them around there are I think So Jimmy has one I'm pointing it out. No, Jimmy has access to one and it's only because he gave up his firstborn, right? They're really hard to get it's it's almost impossible
01:00
So therefore I'm here to tell you how you can develop for HoloLens anyway, and that's why it's cool So I don't need a HoloLens. So if you thought I should have a HoloLens you can give me one No, okay. So why am I here? My name is Lars and and I am actually Danish But I have lived in the land of fruit and nuts. No, not California Australia for a very long time
01:23
Which is kind of why my accent is like this. So I'm drink the kool-aid I talk a lot about Microsoft Windows platform MVP categories where HoloLens belongs for the moment So that's kind of cool because I fit in there. I do some pearls like courses Which have been a bit of mix kids cause some Windows Phone stuff. Yes
01:43
I am the guy that uses Windows Phone in Australia and I've been Troy's test subject on a couple as well So if you don't know what pearls that is or if you would like to try I have some free passes So feel free to grab one after the talk I'll write a bit about different things here Also about HoloLens, although I haven't written much because it's so new
02:03
And I just want to mention this. Did you know that NDC is going to Sydney? Yeah, so yeah. Yes says the Australian So if you want to have another go and a trip to Australia because we have really really nice beaches We have beautiful fauna and we welcome everybody equally
02:24
We We really like you to come and obviously someone else likes would like that too This is generally the perception of Australia It is not true. We are quite a civilized bunch as long as you provide us with beer Now I have this
02:42
Experience from last year at NDC and and I had a Norwegian audience That's the thing. Did you know that I had it's like we use in the world It's like oh, did you get Norwegian audience? Oh, no Please don't be like this. Please don't be these guys The best way to learn is to do this
03:01
Ask questions be interactive. I don't mind. I'm I thrive on questions. Okay, and throughout the talk If you follow this hashtag, I'm gonna tweet whatever is relevant in the talk Hopefully at the same time as I talk about it So that's just a bit of context if you miss out on a thing or a link or a video or whatever it might be
03:21
Okay So enough about me Let's talk a bit about what we do every day. So the current state of software is that we develop things For a screen, is that right? How many here are software developers? Okay, everybody else Anyone here is a manager or something?
03:41
No, good out, you know now this is for everyone But the thing is we all live in this software world and we more so than everybody else we have a Duty to provide good software to users. The problem is though that we are limited But what we can display it on so if you're a mobile developer you develop for you know a flat screen
04:01
It's in someone's hand granted. You have lots of You know accelerometers and GPS and all sorts of sensors that you can use but you are limited to that screen, right? It's the same if you're a software desktop Desktop software developer, you can only develop for whatever people can show on the screen And it goes on if you do games does anyone here do game development by the way
04:22
Do we have any game developers? It's like one and a really really long arm. He made Donkey Kong Because that's an advantage in whole in holographic programming no question, especially because the 3d aspect But with games you still you sit down unless you are using a Kinect or Wii
04:40
Which I haven't seen anyone do without injury Then you sit down right and you limits it to the screen again and the same a collaboration And this is a really cool point about HoloLens collaboration But you normally just have I don't know if you sit and point at a screen that seems a bit odd But there are very very physical limitations to what you can do. Yeah, come on in guys
05:02
If you sit up the front, I'll point now. I won't point you out but the gist of all this is that When you've been doing it for 20-plus years, which is why I knew all the games yesterday, oh god It kind of you kind of become a little bit of a boring duck It all becomes a little bit the same right and you just sort of go another one and you got oh this one
05:24
This this thing I can use GPS. Yes And it's sort of you try to start focusing on the little things that make it worthwhile, right? So I'm in the same boat. I do mainly web development I'd love to do more Windows apps, but no one gives a shit
05:41
Unfortunately, so that's sort of the current state and I guess I'm here to try and tell you that it can be really really different And you can start right now, even though we don't have these beautiful devices So let's just I know so just to show hands again because we like like double of book before if anyone were at Renee's Talk yesterday, please just show it. Yeah, there's a few there. So and that's okay
06:04
I'm gonna try and do a different take on it There will be some overlap, but I don't have a whole lens. So I have to do it differently, right? Okay Just so you know what you're in for so if you don't like that out no you can say The first thing is virtual reality. So we have this concept of virtual reality, which has been around for god knows how long I
06:24
Was I think it was eight or nine when I did virtual reality for the first time and he was in I don't Know in this contraption thing and you're holding Like a trigger that would make you move and then you're supposed to shoot the other guy and it was all 8-bit and it was All it was kind of like the demo yesterday. It was all squares and spheres and that was it, right?
06:41
So it's been around for a long time, but it is a complete virtual world So everything you see is replaced Everything you see in this world is made by someone else It has very limited physical movement of it because if I had an Oculus Rift on now, I would walk off the stage So you can't really move around because you can't see where you're going
07:03
But it's great for prototyping if you're trying to show someone what a new building is going to be like There's nothing better than give them around the virtual reality and they can look inside the building, right? So there are really really good Use cases for it Next thing is augmented reality. So you probably have a smartphone. Yes. Anyone doesn't have a smartphone
07:20
Because I don't want to see it now. It's so prevalent right? We all have these devices in our pockets, which are mini computers Augmented is Kind of like a tiny bit of what Hololens can do. I guess it's the best way of explaining it. It's usually triggered by Some sort of object and your phone will pick it up with a camera and it'll show something
07:42
It'll augment the reality. Usually what I've seen is brochure wear. So you have I'm gonna buy a boat I plant my smartphone app at this boat and the boat comes alive on the screen, right? It comes 3d very cool It's really powerful And then because Hololens didn't really fit into any of these sort of digital realities as I like to call them They had to come up with their own. So it's called mixed reality
08:03
And I'm sure this may change in the future as people find out more about what it can do But right now the term is mixed reality and these are all digital realities It's just different ways that we are interacting with a digital reality So the Hololens the whole idea is you have seamless integration of real and digital As we if you and Renee's talk you saw this you saw that, you know, it'll interact with whatever physical devices you have next to you
08:30
The whole idea is to have natural interactions Right. So whatever you do normally, you know, it's still getting there I wouldn't say it's still normal to do this all the time, but we get in there
08:40
It's an interaction that you can do. It's a physical movement that you could do in real life And then the whole idea is you can use any room any surface anywhere So normally when you look at virtual reality and Troy kind of touched on this in his keynote You get this feeling right? Like this is usually the reaction people get it's like holy shit
09:02
Because it's it replaces everything and you don't have your inner ear balanced necessarily. You don't know what's coming You can't escape it other than just scream, right? and it does replace your entire physical reality to something that is You know much more real I guess I love that one
09:20
So that's what sonic looks like in real life But hololens to me is is it's why we're here and it is just absolutely pure magic Like I tried it on I have used one Um, and it it does blow your mind even though you've read everything about it. You've seen all the videos. It's just that whole um Change of what your world is is quite impressive
09:45
um one important thing that I have come across though the more I read about this the more I use the tooling the more I Try and come up with ideas that are really um going to blow people's mind as we always say We need to understand what is the difference between a hololens app and a virtual reality or augmented reality app
10:05
and it's an important difference because I don't want to take any renee stunned that way because the flight the holo was a holo flight yesterday was absolutely impressive, right? But to me it wasn't a hololens app Um, and I might be a bastard for saying that but it just wasn't Because it wasn't interacting with the real world the planes if they could crash maybe into the wall or whatever
10:25
It might be but that would just be you know, morbid It wasn't a proper hololens app. It was impressive and it was really really polished and well done. No question about it But something like another test or or um, not test but demo app that microsoft has is this game
10:41
And in this game, which I forget what was called. You know, what's called. Um Fragments which is fragments of memory for this boy and trying to solve a mystery what happens to the boy And you look around the room and things gets replaced depending on what room you're in And people will come into your reality. They'll sit down on your couch and talk to you like that to me is a holographic
11:02
App it's hololens app. It changes what your physical world looks like Important distinction at least to me anyway, okay Let's have a look at some of the hardware specs I love this picture Because all we've seen is on the right we've all we've seen is that hololens, right? And it looks it looks pretty good. It looks a little bit like you're from x-men, but it looks pretty good
11:23
But it came from the other one It came from that prototype and that's the earliest prototype I could find any pictures of And the thing he's got on his head that black bit is actually a Kinect So he's got a Kinect on his face along with all the other gadgets So just bear in mind we've come a really long way
11:40
I'm not sure. How long does it take? I think it's about five years so far five six years project. So it's it's not simple So the specs for this they are important as much as we like building software I'm not a hardware guy really as long as my laptop works that it doesn't hold up. I'm pretty good But it's important to remember what the what is inside this thing
12:01
Um, so that's why I say hardware and then I'm gonna say Windows 10 Which is not hardware, but it's important to know that Windows 10 runs on this device and as Renee touched on yesterday as well UWP is how you develop apps for this So you use the universal Windows platform and you can use if you're a masochist C++ I'm not man enough to use C++. So I use C sharp
12:21
Just because it works everywhere and it it does a lot of the heavy lifting for me But there are there are scenarios where you have to use C++ just because of the performance, right? So I'm just walk over here because I like you guys, too It's an Intel processor, so it's 32 bits and people are not 64
12:40
I don't think it matters Often 64 just means that it takes a little bit longer to process because the memory space is bigger and you need a more powerful Processor using more battery and there's all lots of drawbacks as well. So I'm not too worried about that. This is cool though HPU anyone heard of a HPU before no a holographic processing unit Yeah, now we're talking x-men, right? How cool is that?
13:03
But it is the thing that makes holograms real it is the thing that calculates all these millions of polygons that fast And I don't know there hasn't been any specs Officially announced about it, but rumor says it's about a terabyte a second of data that it processes So it's yeah, it's pretty serious thing
13:22
Two gigs of RAM which doesn't seem like a lot. I thought it'd be like with today's RAM We're like 200 or something, but two gigs of RAM 64 gigabytes of storage is where you install your apps So it still is a fully self-contained computing device, right? Has four mics So the mics above your head essentially if you saw on the on the device and it's really good. They're really really good
13:44
I'm very impressed with this speech recognition in Katana in Katana. It's Katana using it but in the HoloLens And then four speakers which sits around your ears, but I'll get back to that later And then Bluetooth Wi-Fi as well But the one thing that is the most important thing take away all this
14:01
googly guard here is The fact that this thing is untethered Like there is no wires off It is a completely self-contained unit and that to me is now it blows everything away Because you can walk you can put a hologram here on the wall So that's in your in your living room and that might be you know Netflix streaming or something and you say to a Netflix stop and it'll stop
14:21
And then you walk in your kitchen make a cup of coffee you come back and it's exactly where you left it Like you don't have to remove the hole and you can take it off and on again if you want but things Exactly where you left them if you're a good developer Because it's not that easy There are lots and lots of abstractions and tools to let you do it. But um, it can be a little bit
14:42
Let's say daunting to read all the documentations on all the various things that make a hologram stick But I won't go into that because it's boring This is normally what you see Have you have anyone here followed HoloLens sort of marketing blurb and tweets and Facebook stuff and seeing the videos and all that sort of stuff Yeah, yeah a few nods. Please don't be Norwegian. Come on
15:03
There's a fear. So you've seen this and this is normally what you get you say oh this guy's staying there and look he's got the ocean and it must be Australia because the sharks about to eat someone and This is actually not how it's gonna look because the field of view and this thing is very very narrow It's more like this so it is like a box in front of you, right and
15:23
That's why we're talking about sound later because sound becomes really important You can't see much and that's why every single demo you see they kind of look like this Because they can't see anything right there they're trying to look even the whole hologram can't fit inside that field of view sometimes and
15:40
It takes a little while to get used to maybe an hour or two But it is a limitation for sure it is something that you need to be aware of because people can't necessarily see everything they do and The other thing is you put a hologram over there and then you saw Renee yesterday He just fired off a hundred of these meatballs if you had put a hundred things different How are you gonna remember they are like you have no idea, right? So there is that as well
16:03
You can't see necessarily you forget where you put things All right, just to blow your mind a little bit. Let's have a look at at this video. It's from e3 If I can get over here Which is a gaming conference, oops, I'll put sound on sound please
16:27
The sound on I'm not sending any sound. Oh, no. All right. I'm gonna do that again. They're more gods Who still nothing? Oh
16:44
That's really disappointing that worked about five minutes ago Hang on this is plugged in I'll get a notification for nothing. All right, bear with me guys
17:14
Cuz it's important I do need sound later, so let me just get this sorted Okay. Yeah, I did that one. I think you see I'm not a hardware guy
17:43
Aha, thank you Okay, right click I really don't know what I'm doing
18:00
Yay, Oh problem All right. Let's try again. I didn't enable the other one Well, I could do it again Really Okay
18:25
Touching welcome to my talk on hardware configuration what I want All right, let's keep going so This is from e3 and this was one of the earlier videos that of demos that Microsoft did on stage
18:42
Sax grabbed his xbox controller to play through a giant holographic portal into a 3d minecraft world right on the wall Even though I don't have a hololens I'm able to play together with sacks from my surface Are you gonna come join my game Lydia? I'm in now. I see you jumping around
19:05
One of the cool things about hololens is that it lets you display minecraft virtually anywhere You can get different perspectives on your minecraft world It's awesome to play with the controller, but could we show them something new sacks? Sure, let's take that experience off the wall and then put it on the table over here create world
19:49
Sax can easily navigate and Manipulate the world using his voice and his hands He can walk around the hologram pan around for different viewpoints and even look inside
20:06
How cool is that? Holy crap And that's why they use minecraft rock because we everyone we all know it and we all know it as a thing on a Suddenly they make it 3d in real life So this was an early example things have moved on since then but I still like this because it really showcases
20:21
What is possible in a way that we can relate to? So, let's have a look now and it's any questions, by the way, I'm gonna keep asking, you know Okay building blocks of of how do you build a hololens app? Because there are elements that are really important. There's four in particular that you need to pay attention to so we got gays
20:42
Gaze is what you're looking at gaze is essentially How you select things as with a mouse over the mouse, so it's your mouse cursor And it's not your eyes So you don't have to program for better people looking it's the center of the hololens as a ray So it's car usually called raycast
21:03
and that's because the object that you use in programming as we'll see in a second is called raycast and The gaze is your mouse pointer for whatever object you want to interact with so it's important, right? It's important that you know what people are interactive wanting to interact with
21:20
so Let's have a look just quickly so I've got here's something I prepared earlier. Can you all see that? All right Yeah, I know it's a little bit tricky in this room with the middle I mean, I feel like I'm in a tennis match, right? I'm doing this all the time But you guys just try and see if you can see a screen, okay
21:41
So all hololens apps as our hololens objects Are built in unity actually, we can start in unity just for good good measure. So you set up a scene Which is essentially, you know your app. So as an app has at least one scene And I have some 3d objects here. So this is my cursor
22:04
Here so if I double click on that You Get lots of lots of information about it. And if you don't know unity, don't be scared. I had like six months ago I had no idea what any of this meant. No clue. It is not that scary Most of these things you never have to touch or have to touch once
22:21
But you set everything up in unity in this case We have this cursor Which is just a 3d object of a ring and that ring we want that to be where people are looking, right? We want that to be the indication that you're looking at something and where is this placed? So the way you do this is you create scripts So down here, I've got folders and I have some scripts in these folders and they are just C sharp scripts
22:43
If you know C sharp, it's really is as simple as that We create a script by going through all the you know, far new script blah blah blah And then we drag and drop that script onto the cursor and this is the case. It's the world cursor, right? I just want to show you the code because otherwise we can spend really a long time. I'm just going through all this
23:02
Every single script that wants that you get attached to objects have a start And an update start is basically where you set things up that runs once on Initialization of the app and update runs every frame. So there's 60 frames a seconds on HoloLens Without exception everything has 60 frames a second because that's the way you make people not throw up
23:24
So it's really important that you have really really good frame rate And there's lots of tooling to help you do that But all this does is that we have a Mesh renderer, actually, that's the mesh for yep, and in here
23:41
We have the transform. So that's where the camera or HoloLens is facing the position So that's your head position whether you do this and then we're forward. So that's where you're looking. What's the raycast, right? And Then you use this physics dot raycast, which is a boolean Give it the position of where you're looking and if that's true, then you can use this hit info
24:03
Which is a raycast hit so that's what am I looking at? Right, this turns false if you're not looking anything or if the if the little ring doesn't hit anything, right? Which is why we have enabled false pretty simple if we do hit something we enabled the the cursor the mesh renderer and
24:23
Then we set the the hit info so the point of that is So the hit info of the raycast and the point of that is what we set the transform position to be So that's the way we're looking and then we can rotate it So the position of the mesh, so this is complicated the position of the mesh compared to where we're looking, right?
24:45
So if we're looking as we've seen a minute at a ball We'll put that on the ball and then we have the rotation here, which was saying Well, let's make it hug the surface So it looks a bit more real so you don't just have this flat ring that sort of sits You know not in like in in real life. So the way that looks
25:01
Is if I just go in here We'll just debug it without debugging So this is the emulator, right? And that's a really really cool tool. It's actually incredibly well done for a first-gen device The emulator does everything to HoloLens does Wallace obviously without the physical experience of it, but it does gives you
25:25
gives you 3d space it gives you all of the Yeah reactivate Has all the tooling goes with it so we can see where everything is a little bit more
25:43
So we have all the coordinates of everything that you would be looking at. So this is your head coordinates Once it's not starting come on So that's the start menu I've just press start and I'm just gonna go over here and I'm gonna select my origami
26:03
So the tap is what I was doing. That's just the enter key So it's as simple as that you just have enter and you have stopped and that's that's pretty much it So here's a little app. This is a And I'm you're using the arrow keys and the gamer keys WASD and I'm can move around this and as you can see when I go over the surface here. Hey
26:26
Just zoom in a bit or move in. That's our ring, right? So we're hugging the surface as we go around And that's it. That's all you do. I mean, obviously you can have this cursor be anything You can be your company logo. It can be a finger could be whatever you want it to be, right?
26:40
So that's gaze. It is really as simple as that Okay, the next thing which is probably a little bit more interesting Is once that comes up, you know Gestures, right? So we want to be able to interact with these elements
27:02
Want to be able to interact with the objects that we're looking at and that's where we use gestures the most common one of Hello. Oh, you're still here Now the most common one of these is the tap and there's what's called a Gesture Field of view or gesture frame and that's essentially where the whole ends is looking for your finger
27:23
So you can have actions based on whether you it can see your finger or not But essentially is all is this you just do this all the time. There's a whole lot of this and that's mouse clicks and It is meant for you to do this. So if you move your head, it can't see your finger, right? So you you start becoming a robot because you got to sort of move your hand with everything
27:43
So it's a bit mechanical, but it works really well. It does pick it up really really well Okay, so that's the tap gesture it also comes as tap and hold or Tap and hold and move in 3d space so you can move things around manipulate them And then you have the bloom which is just that that's just start menu. That will always work
28:03
You can always get the start menu up And again see there's again more marketing shots No field of view displayed at all So let's just try and make a few gestures, shall we? So the way that you do gestures is
28:24
You have a gaze gesture manager. So that's the It's built in the SDK So all these tooling by the way are free and if you follow the last clint twitter handle I will eventually tweet a link to where you can get all the tooling and how to set it up and I wrote an
28:40
article yesterday on on the whole introduction to building HoloLens apps so We have a game as a gesture recognizer, which we actually recognize the gestures and We then on start which is our initialization, right? We attach these events event handlers. So tapped event comes for free
29:03
We're gonna do that a lot, right? And all that does is that if you if it detects a tapped event Then we say well if we actually looking at something then we send it an on select action Right, and then we just start capturing gestures and that shall start and then every frame
29:22
We do a raycast again and we set the focused object, right? So every frame we get the raycast and we look at okay. Are you looking at something? No, yes No, bull bull bull bull bull bull bull whole time 60 times a second Which means that when we actually look at something when we tap it knows whether we focused on something or not, right? And that is really as simple as that and then yeah, we cancel gestures and start capturing them
29:45
bit of Optimization there so we get our right frame rate and all that does So again, we up here. I'll just click on my emulator so I can move around here So when I oh, sorry, I forgot to show you what actually happens
30:01
So here on select That's that method here So that's attached to the object itself the thing that I can select the thing that I tap on We'll get this so if you tap on something doesn't have the on select well, nothing happens. It's just an event, right? Does this look familiar C sharp developers? This is something you could do right? Well, you can read this
30:23
It's not rocket science. It really is quite straightforward Because initially when I thought a whole ends after like oh god It's gonna be tricky because there's maths and there's 3d space in there. Ah, but no it is Initially very simple like this as Renee said you do get into situations. Sometimes that are not
30:41
All this does is it says if you don't have a rigid body that essentially just means do I have physics So does it 3d object have physics? If not, you give it physics So what happens is that when you tap on it, so I'm just gonna hit the enter key here falls down Cuz I'm in the in the room. Actually, I'll show the room
31:03
And they're suspended in midair because they don't have physics. They're like magic, right? One thing I just want to show you that now we're looking at the emulator, so there's all these things here There's all you know, the body was that moved if I tilt the head So it like so if I move here, so Q and E will tilt the head So you see my head can roll because you could do this right when you have a whole inside
31:22
So you can that you get that information as well. You can select what room you're in So their rooms pre-loaded if you have a hollow lens, you can record your own room and put it into the emulator Makes it even more useful But all this is that has this whip our device portal which is a website so this is now connected to the emulator and if you had a
31:43
Physical device you would get the same information just for the physical device And the really cool part of this is this 3d view So this is my plane. So I just use the mouse key here. It's going to look I mean obviously this doesn't look that impressive because it's just a bunch of squares if I update my surface here It builds the room for me that I'm in
32:03
So I can fully completely see where this room is just a bit like that Move it over here. And then if I open the emulator Right as I move around See it moves over there as well, right? So obviously I would have this on a different screen. I normally work with three screens
32:23
Just because there are so many tools that you want to see at the same time So you get a fully I want to I don't want to say immersive but of a full experience of what you you would experience in a HoloLens, right and There's many more tools in this especially around performance Which is a complete talk on its own because it's it's a reasonably complex topic
32:45
If you want to have lots of holograms, especially at the same time, even if they need to interact as well So still no questions Is this yes, yeah
33:12
Do I know anything about rolling out consumer versions is anyone here from Microsoft No, I don't know a thing I have no idea no, I actually don't know
33:21
There's been nothing nothing that I've heard at all right now It's deaf to deaf kids and the one I tried is very much a deaf kid. It's like a prototype essentially, right? Quite fragile they do work fully as you see in the videos There is no there's no magic there in terms of photoshopping But it is you know The field of view and the the way that it sort of feels is still very much a prototype but consumer versions
33:43
I don't know I can guess Two years three years. I don't know. I think it's probably that far But it's I hope it won't be three grand For one but no, I don't I don't know when the consumer version would have would be released Yeah Mm-hmm
34:02
So is there any plans of expanding the field of view? I really hope so That's the only I can there was a job at someone posted that was announced for Microsoft of a an optic specialist or something For the in something devices unit or something. So I mean, hopefully that's a good thing, but I would have thought so
34:20
I think it has to do with processing power So once they get the hardware better, they might be able to do more but I actually don't know Again I really hope so because it would make it a lot more immersive for sure Okay Let's go back and look at the thing that I really hope it's gonna work bit temperamental sometimes so
34:45
Voice is quite important because you can't keep clicking on things, right? You can't keep doing this because it gets boring and you get tired So you can talk to the device and it'll understand everything you say I mean Thomas here would dance because of you. Are they all who they?
35:04
Men I mean, but but in English, it almost doesn't matter what accident you have I think if you have really like strong Scottish Irish or something, it might be a problem, but it actually does work really really well Okay, I'll get rid of that guy for sorry. Um, there's a there's a command which is select So you can always look at something and say select if it is selectable
35:24
Something will happen and that says you can't override that it's a you know, it's a reserved word select will always work And of course katana works, right? so we can talk to katana and say Cuz it's gonna bleep everything over here But you say no katana and she'll come up and she'll you know start working on your device as well
35:45
So yeah, it's internet connected And then there's custom commands, which is by far the most useful thing And there's lots and lots of theory around this that I'm I Won't go into as such But just bear in mind that sound or speech design is
36:04
By far the hardest part of this getting someone to do something when you say something is not hard, but having a whole Catalog of commands that you want to your users to use is by far the hardest because you can't have words that are Too similar. You can't have sentences that are too long because people don't remember
36:22
you can't have So what is it? Yeah, you have to be non-destructive, right? So you can't have things that override other things so that if you have Words that are similar or you know, they don't override each other You shouldn't have more than two syllables in words because then it starts getting hard to recognize it or harder
36:44
So there's all these guidelines around how you design your speech or your your voice commands and that's the hard part So I want to show you the easy part Which is Really really quite simple and I was surprised because this is I've done a fair bit of katana integration
37:03
Has anyone here done Windows apps? Is there any Windows developers? There's a few Hello Have you used katana as anyone here use katana on an Android on a you know on an iOS on a Windows phone Whatever might be oh, come on. It's like the good version of Siri. Come on
37:23
No, and I know what you're saying because it's it's a thing we're not used to it's a Unnatural thing to go around and go. Hey katana. Tell me a joke, you know, it gets boring quick By the way katana has told over half a billion jokes now someone does it
37:42
Sorry I Told you she was gonna wake up. All right, I can show you but first I need to quit the current app Is that okay? No All right I'm not even sure what she was gonna quit
38:02
Wow Calm down So let's have a look at speech And and it is really simple. You have a keyword recognizer. I shall go down the bottom here first Oh That's another. Yep. So you have a keyword recognizer and then you add keywords to essentially an array and
38:23
the keywords are text strings So we're back to magic strings again, but they are just text strings, right? And that means you can do whatever you want. So we have two here We have reset world and we have drop sphere. So they're different enough, right? They're short. They're different. All right fair enough
38:43
And then we with these keywords we attach them in here to our keyword recognizer and then whenever it Recognizes a keyword it fires this unfreeze recognized And we get all this for free. It's all built for us. It's all part of the SDK and Then we have our event that says unfreeze recognized and we invoke whatever that lambda function that we gave it up here is
39:06
Right. So there's other ways you can do it as well, obviously But that's the gist of it and by the way all this code here I've blatantly stolen from the Microsoft tutorial because I want you to do it, right? I want you to go out and do this if you have any interest in holographic development
39:22
It is that simple and again follow the Twitter handle and you'll get with all the links are And so what this does is that reset world sends an on reset event and Drop sphere sends an on drop event This is a broadcast message, so that's actually for every single thing not just what you're looking at
39:42
Whereas the drop sphere is only for what you're focused on, right? So that's a slight difference here. We'll go back to our App here. All right, see if I can find It's the problem when you are there we go. So there's my other ball here. I Need to get my yeah
40:02
Did you remember that I dropped it to the floor that right one now, it's back again It's because I said the keyword and it was still listening. So if I drop this And then I go and I do the same with the other one Actually, no, I don't want to tap it. Right? We just made in voice command so I can say drop sphere
40:24
You know falls down, right That's that's that's how well the emulator works because my pronunciation is okay, but I'm certainly not British, right? It's not super clear I picked up the ain't gone. Yeah. Yeah, right the Australian part of the things, right? So
40:40
And then the other thing we could do obviously is that we could say reset world Hey, and they come back and you can have so much fun with this Like you can keep making up, you know keywords and do things and make them dance or whatever you want to do, right? It's the actually implementing the keyword part of it is not hard, which is my point Everyone here can do it. You can it's really not hard
41:03
So Let's have a look then at the last of the four Pillars or building blocks that you need to know so we've gone through gaze gestures and now speech So the last one Is sound right?
41:22
Sound is very important because as I said you might have people put Holograms here there and everywhere and you need to know or you need to let your users know where these are if they can't find Them so having sound especially spatial sound Like this what Hollins does? is crucial
41:42
And to be honest, I haven't mastered this it's really hard To get sounds that are appropriate You don't have to worry about where the spatial sound bit of it because the SDK does all that for you but this thing the This the immuno to show the SDK does what's called the head related transfer function, which is very clever
42:01
I'm sure but all that means is that it mimics what your ears would normally hear So a sound the speakers above your ears on the HoloLens will put out Sound that is slightly adjusted to each based on where the direction of the sound is so if you if I heard something over there it would reach this year later than that year and that's the meme of the simulation or the
42:23
I guess what the HoloLens does to make you turn your head and say oh it's over there, right? Very simple concept, but really really hard to implement, but we don't have to which is good So, I'll just see if I can get this demo to work as well So does that make sense?
42:41
Those four things is what you need to know, right? Those four things is everything you need to know to get started I'm not saying is everything to know ever But it is to build relatively simple apps for HoloLens and get you started that's all you need to know so sphere sounds So these are assets in
43:04
Sorry in in unity as well, so it's small anyway then he hopes Here we have an ambient sound so that's a sound byte so that's part of our project, right? It's an asset we have in this project
43:22
I don't want to go through how you build and all that because we're gonna do that yesterday but building and deploying your HoloLens app is a little bit tedious because you gotta go into unity Export it to Visual Studio open in Visual Studio start up build it, you know run it on the emulator wait for it and so on
43:41
Rene was a bit clever about it He had some scripts that would sort of alleviate some of that pain and make it a bit easier But it still is a bit of a long long haul So let's go back to Visual Studio So we have again on start we set up our audio source and down here we have two clips impact and rolling
44:02
So we have when something hits and then we have obviously when the balls are rolling so Naming remember Jess's talk last night the hardest thing is naming in computer science, right? So try and make something name something that makes sense so on collision enters that's when we hit something with the object
44:25
We then go and we calculate if we should Play the audio clip. So that's the relative velocity. Is it you know, does it roll fast enough? And then we have the Uncollision stay so that's when it stops and then we still trying to figure out what has it really stopped
44:44
Those are still going something and then based on that we play a video clip. So video audio clip or not And on collision exits, we just stop the audio source. So again relatively simple code The main thing again is to try and find audio clips that make sense
45:06
Okay, now these didn't actually play audio which I thought they would have So I try again and now it's frozen on me. Hooray
45:20
Yep, I'm pressing buttons Yay. All right. Trust me. There is audio on it. All right, I did I had to have one demo fail, of course But I'll just keep I'll keep going because the audio is you click on some of your falls down It makes a noise. These noises are sort of you know a bit crude but
45:41
We what I was going to show you was that when it rolls down you turn and I turn the the emulator The sound changes as well. So I haven't done anything to do that I haven't done any code to say Oh when you turn your head left make sure if the balls were on the right and blah blah There's none of that. It just comes out the box, right? Okay
46:02
So please any questions on any of this because that's kind of like that's your super fast introduction to HoloLens coding If yeah So how is it to build things like web views? So you're talking can I read reword then say how is it to build 2d apps, you know?
46:24
Is that essentially what you're asking? Yeah, so 2d apps work really well Because you can essentially pin them on the wall, right and you anywhere can become your screen You can have them suspended in midair as well. And as you walk around them and see the backside it's mirrored, right? It's kind of cool. But anyway It's not very cool. We need to work it work on it
46:41
So you can absolutely make 2d 2d apps and Outlook as a mail So the windows mail app just called mail now works on HoloLens. There's a HoloLens version of it And there was one more thing. I can't remember I think it was for the outlook or something you guys might know But there are Microsoft is building their 2d apps that work on HoloLens. Absolutely and it's easy because they're all
47:04
UWP they're all windows universal windows platform apps so you can take all your code Almost and just get it and deploy it on the HoloLens and have it in the HoloLens store as well So, yeah you can absolutely do 2d apps and I think there'll be lots and lots and lots of those because they're much easier to build and
47:24
Why wouldn't you have if you have a 3d modeling tool, you might need your outlook there. It makes sense, right? So what's next This is very early days like none of us even have a HoloLens yet We're all here. We're all trying to find out what it is involved in building one
47:43
And I hope it's not gonna be like this Because that would just be really awkward But yeah, there are some social challenges because the truth is when you put one of these on You can't help but look a little bit like a dick
48:02
Like it's I don't think it's a good look and because you see everything else that no one else sees You have this big grin on your face and you're starting Yeah, it's not a good look so there's some social challenges for sure, but it's very early days, right? No one has one in it. If we see one it's because well, we geeks and we go. Oh, it's cool We don't kind of think through the things that you know, a mom will go really
48:24
Actually, you know So the challenges we have is to build software that's gonna make sense and not be completely unachievable like this the Mars Rover HoloLens experience I could build that I don't have the mass data
48:41
Like that just seems really far-fetched, but we need that to be inspired to build things that I'm like, maybe not as complex Right. I think it'd be more like some like these sort of tool bench type apps So something where you go and you say okay I can put my toolbox here and I can build things in real space and actually move it over on this table because it's
49:00
in the way You know so you still have the HoloLens experience that I'll talk about earlier make sure that what you're building actually makes sense as a mixed reality app and not just a virtual reality app So that's I think this is what's going to happen a lot as developers get the hands on this I don't think it's going to be as Professional as what Renee was building the whole holo flight type app because that's hard
49:24
That is a really impressive piece of software So that's my theory I guess but the real cool thing is that now is your turn right Because all of these tools are free everything I used here was free visual studio community edition unity for hololens
49:41
There's a dedicated hololens SDK for unity the emulator all of these tools are free and I would recommend though to find someone that knows 3d modeling because I have no clue. I am NOT a 3d designer I'm not a I can't draw anything So I'm relying on the unity Shop so lots of 3d models you can actually buy or free ones from from the shop and there's many other ways that you can
50:06
achieve or acquire 3d models as well and basically let you imagine go nuts and your imagination go nuts There's no limits almost right because it's all virtual It's a 3d world that you can build but you can make an interact with your real world
50:22
And then we're speaking about devices before and they are speeding up So now wave 3 has arrived waves are how they're inviting you as a developer to spend $3,000 on buying a hololens And it's it's tricky because I was speaking to various people of oh can I just have one here and someone can hold it and just show it and
50:41
The kind of sentiment is well would love to but if it breaks I can't get another one for one There's no warranty on it and I can't ensure it So it's very sort of if you have one you get access to one when you you still a bit privileged, right? And that's unfortunate, but that's why we have all the tooling So, but they are shipping wave 3. I'm on wave 4 as I said before so hopefully I'll get invitations
51:02
Just you know get another mortgage Bit like that But I want to leave you with something that's hopefully will inspire you a bit because we can't all be super fantastic 3d designers and modellers and I mean most of us have a job as well, right? We need to actually make a living and this would be something extra
51:22
so one of the guys that did acquire a Hololens dev kid thought okay. What can I do? Sorry. I didn't catch that She's very temperamental today So a developer that did manage to acquire a developer kid Built this I just want to show you this. I think this is very cool
51:45
What I think I'm going to do. I think I'm going to just simply put a sphere in the middle of the bush And Then just go around it and see what areas are outside the sphere and just trim them and
52:07
I Don't know. Let's see what happens. I'm gonna try it out and show you guys whenever I get it done So this is what it looks like after trimming I
52:21
Think I did a pretty decent job Let me show you without the sphere which by the way is low Polygon, so it's not quite spherical. So I it's not that great And I could you know, keep going if I wanted to but that's good enough
52:42
So this is what it looks like. I think It's pretty good. I Just needed to grow a little bit more so I can trim it a little better But I'm pretty happy with it and that's how HoloLens Go ahead can help you do gardening, right?
53:04
How awesome is that and it's simple like the implementation is really really easy Create sphere. That's it and you move around a bit maybe but this guy had an idea, right? I mean, I don't know now if I can get my mom to where I want to do that, right? That's probably the challenge But it doesn't have to be a master over landing experience that you're building
53:23
It doesn't have to be a hollow flight type experience like you can't do stuff. That's really powerful Without even having 3d modeling skills. Just creating a sphere But then the other side of things is that obviously we need more inspiration. We can't all build gardening apps So that's not enough to get us going perhaps
53:40
So this is Microsoft research It's called holoportation and I'll probably just let the video Speak for itself because it's It's pretty impressive People play come on. Hi today We're going to show you an exciting new technology that could fundamentally change the way that people will communicate in the future
54:04
Imagine being able to virtually teleport from one space to another in real time Hey Sergio, how does it feel like to be holoported? It feels great to be holoported So Sergio is to wear his HoloLens device And I'm gonna wear mine we can see each other in full 3d in real time
54:24
We can interact and communicate as if we're co-present Sergio. Can you walk around my space you walk behind this chair? So we're doing everything to give the impression that Sergio and I are present in the same space Sergio. Let's just do a high-five
54:41
That's great, thank you Sergio we call this technology holoportation That's freaking awesome Wow Obviously, this is not for us necessarily because there's about a squillion corona worth of equipment around them And they have an internet connection. That is certainly not in Australia to manage all this data
55:01
So there are limitations, you know for us, but that should you know The idea with this is to give you inspiration and what is possible Now before I just go on the last slide any any other questions Yeah, so the question is that we saw how we can place things on physical objects
55:29
But there's something that tells you where the physical objects are Why do you need to know that? Alright, so if you want to have your app just like we saw with the Minecraft example
55:41
He obviously looked at the table because he was very focused and went great world And that's so the HoloLens will see that there's a surface and then it's up to your app Well, you're raycast to see well and what I'm looking at is that Adequate to what my command is I guess So you need there's a bit of logic there obviously to make sure that that surface is either big enough or adequate
56:04
But you can have surfaces on an angle as well and things will fall off Right, if you give them you can anchor them as well and there's all sorts of stuff you can do But you shouldn't need your app shouldn't need to know where physical elements are as such because it should work anywhere Right shouldn't matter what the room is
56:20
What do you app should do is be able to say ah, I'm interacting with something in that room Is it possible to do whatever the users asking? I think that's probably a better way of looking at it. That makes sense Yeah, anyone else? Yes, sir. I had pitch. What was it pitch roll and heave?
56:49
See, I don't know enough. It's a the question is there was was this in unity? Long time ago. Okay, but there was some way I'd said pitch roll and heave I think and I don't know I haven't actually seen that so I don't sorry. I don't know. I'm just like you guys
57:02
Yeah But I'll look it up. I want to find out now. Not right now, but anyone else Wow Almost Norwegian before I leave you. I just want to mention pub conf tomorrow night at Pub conf dot IO is all the details. I'm doing a five-minute talks with lots and lots and lots of animated gifs
57:23
It'll be amazing There's about 12 14 speakers or something doing five minute talks The idea is that slides automatically rotate every 15 seconds, which would be very interesting bit like slide karaoke But almost not Thank you very much guys. Appreciate your time