Cloud-Scale Event Processing with the Reactive Extensions (Rx)
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 163 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/14025 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Extension (kinesiology)Event horizonProcess (computing)Principal idealSoftwareService (economics)Computer programmingData modelCompilerScheduling (computing)Interface (computing)State observerField extensionEvent horizonVirtual machinePay televisionAbstractionInterface (computing)Multiplication signStudent's t-testSet (mathematics)Point cloudWindowGroup actionMessage passingInformationLocal ringPoint (geometry)Electronic mailing listState observerConcurrency (computer science)Goodness of fitWeb browserSequenceException handlingPower (physics)ResultantDemoscenePulse (signal processing)BitFilm editingScheduling (computing)Streaming mediaQuicksortCartesian coordinate systemScaling (geometry)Client (computing)Slide ruleObject-oriented programmingTrailSemiconductor memorySystem programmingRight angleOpen setDescriptive statisticsCircleAverageArithmetic meanUser interfaceQuery languageMathematicsState of matterTrajectorySpacetimeError messageFunktoroutputNichtlineares GleichungssystemFundamental theorem of algebraOperator (mathematics)Interior (topology)Variable (mathematics)Link (knot theory)System callUniform boundedness principleVisualization (computer graphics)Computer animation
08:16
Service (economics)Computer programmingData modelCompilerScheduling (computing)OvalVideo game consoleCurvatureInformationString (computer science)StrutEvent horizonSyntaxbaumEnumerated typeDuality (mathematics)IdentifiabilityObject-oriented programmingSign (mathematics)Link (knot theory)Process (computing)State observerCompilerQuery languageWeightNeuroinformatik2 (number)Streaming mediaSystem programmingType theorySystem callVirtual machineAddress spaceEvent horizonAbstract syntax treeRun time (program lifecycle phase)SpeicheradresseWalther-Meissner-Institut für TieftemperaturforschungSet (mathematics)Operating systemPay televisionAttribute grammarPoint cloudMappingMultiplication signExpressionFormal languageData managementSequenceVariable (mathematics)Internet service providerGame controllerFood energyOperator (mathematics)Tracing (software)Point (geometry)CodeTelecommunicationSelectivity (electronic)System administratorTwitterRepresentation (politics)DatabaseParsingExistenceAbstractionIcosahedronImplementationWindowLocal ringScaling (geometry)Context awarenessDenial-of-service attackClient (computing)Presentation of a groupExtreme programmingHand fanBootingMereologySemiconductor memorySequelDistributed computingComa BerenicesComputer animation
16:33
OvalSystem programmingComputer programRegulärer Ausdruck <Textverarbeitung>Execution unitMusical ensembleVideo game consolePressureFluid staticsParsingExpressionOperating systemPoint (geometry)Problemorientierte ProgrammierspracheEvent horizonSequelMultiplication signSyntaxbaumQuery languageState observerKeyboard shortcutException handlingFile formatModulare ProgrammierungNetwork topologyArrow of timeStructural loadFormal languageCompact spaceInheritance (object-oriented programming)TwitterOpen sourceInformationSoftware frameworkState of matterCuboidDiagramNumberSequenceType theoryCombinational logicVideo gameBitPlug-in (computing)Game theoryService (economics)Object-oriented programmingTerm (mathematics)Link (knot theory)Semiconductor memorySpacetimeMembrane keyboardWebsiteProcess (computing)QuicksortCodeBoom (sailing)Alphabet (computer science)ImplementationOnline helpMechanism designOrder (biology)Personal digital assistantLetterpress printingFactory (trading post)Context awarenessReduction of orderPoint cloudPhysicalismDemo (music)System programmingDiscrete element methodLie groupLine (geometry)Sheaf (mathematics)String (computer science)Constraint (mathematics)RhombusInterior (topology)Stack (abstract data type)Lambda calculusWeightSerial portBuffer solutionCommunications protocolBootingVariable (mathematics)Translation (relic)2 (number)Run time (program lifecycle phase)ParsingServer (computing)DatabaseInternet service providerParameter (computer programming)GodComputer configurationVapor barrierComputer animation
25:56
Directory serviceComputing platformProcess (computing)Dean numberPoint cloudNormed vector spaceService (economics)Maxima and minimaLink (knot theory)outputTerm (mathematics)EmailThomas KuhnArchitectureModal logicExecution unitDynamic random-access memoryWechselseitige InformationService (economics)System programmingLocal ringMultiplication signClient (computing)CodeWindowQuery languageFlow separationComputing platformSemiconductor memoryData storage deviceMappingPerformance appraisalVirtual machineWordTable (information)MultiplicationCombinational logicData centerBuildingImplementationKey (cryptography)Video game consoleCoordinate systemState of matterSyntaxbaumServer (computing)BootingPoint (geometry)QuicksortProjective planePersonal digital assistantDemo (music)ExpressionTerm (mathematics)VirtualizationAsynchronous Transfer ModeCore dumpPoint cloudPlanningSurfaceDifferent (Kate Ryan album)GoogolFamilyEntropie <Informationstheorie>Insertion lossDatabaseSocket-SchnittstelleQueue (abstract data type)Moment (mathematics)Computer animation
30:09
Computing platformProcess (computing)State observerPay televisionFile Transfer ProtocolVideo game consoleLocal ringTransponderDedekind cutSystem programmingTask (computing)SynchronizationFluid staticsTerm (mathematics)Link (knot theory)Type theoryDemo (music)Online helpQuicksortDatabaseConnected spaceClient (computing)Goodness of fitLogic programmingContext awarenessService (economics)BitPhysicalismProjective planeCuboidLine (geometry)Personal digital assistantNumberRemote procedure callLocal ringComputer fileWeightMultiplication signAuthenticationSequelProxy serverDirected graphField extensionHydraulic motorSurfaceGraph coloringMenu (computing)FreewareComputer animation
32:20
System programmingSynchronizationFluid staticsTask (computing)Term (mathematics)Scheduling (computing)Service (economics)Data modelComputer programmingDialectInterface (computing)Computing platformState observerVideo game consolePay televisionLocal ringFunction (mathematics)outputFactory (trading post)Streaming mediaIdentifiabilityType theoryMoment (mathematics)Line (geometry)Interior (topology)Context awareness2 (number)ExpressionVideo gameEvent horizonString (computer science)HierarchyMereologyDifferent (Kate Ryan album)QuicksortSemantics (computer science)System programmingAbstractionServer (computing)Table (information)Equivalence relationPoint (geometry)Computing platformView (database)Multiplication signKinematicsEncapsulation (object-oriented programming)Message passingAliasingDemo (music)RoutingKinetic energyConnected spaceProxy serverPotential energyLink (knot theory)TelecommunicationEmailIntelligent NetworkSequelRight angleCartesian coordinate systemObservational studyDiffuser (automotive)Semiconductor memoryKey (cryptography)SurfaceState observerData storage deviceSheaf (mathematics)Communications protocolComa BerenicesTouchscreenDirected graphElectronic signatureIdentical particlesMechanism designConfiguration spaceCode refactoringSystem callComputer animation
38:00
Normed vector spaceExecution unitFluid staticsTask (computing)SynchronizationTerm (mathematics)System programmingEmpennageGraphical user interfaceCore dumpNetwork topologyEmailSimultaneous localization and mappingHardware-in-the-loop simulationOctahedronPay televisionContext awarenessSystem programmingIdentifiabilityPoint cloudServer (computing)ExpressionState observerLibrary catalogOperator (mathematics)Video game console1 (number)Service (economics)SubsetDot productQuery languageStreaming mediaSyntaxbaumComputing platformMereologyNetwork topologyPay televisionPlanningEvent horizonClient (computing)CodeNeuroinformatikMultiplication signDemo (music)Table (information)Revision controlCoordinate systemInterior (topology)Right angleCuboidWindowMessage passingProxy serverMobile appSystem callMechanism designProcedural programmingQuicksortSeries (mathematics)Goodness of fitComa BerenicesPoint (geometry)Formal languageBuildingStatement (computer science)StatisticsData storage deviceMacro (computer science)Expert systemHand fanPhysical lawDatabaseSequelCartesian coordinate systemLattice (order)Uniform resource locatorGroup actionDefault (computer science)Core dumpDirected graphGame controllerRow (database)Computer animation
43:41
Infinite conjugacy class propertyTask (computing)Fiber bundleSynchronizationNormed vector spaceDemonSystem programmingInterior (topology)Convex hullDrum memoryPoint cloudWindowVideo game consoleService (economics)TesselationData centerProxy serverExpressionClient (computing)Streaming mediaComa BerenicesDirected graphRight angleState observerDemo (music)Cartesian coordinate systemMessage passingSurfaceQuery languageSyntaxbaumServer (computing)Context awarenessLetterpress printingIntrusion detection systemSequenceXMLComputer animation
45:28
Demo (music)Context awarenessFluid staticsSynchronizationTask (computing)Binary fileForm (programming)Local ringState observerVideo game consoleExecution unitEmailConvex hullSystem programmingSpacetimeRing (mathematics)Design by contractInclusion mapAnalytic continuationInternet forumEmpennageMaxima and minimaKerr-LösungScripting languagePay televisionScheduling (computing)Interface (computing)Client (computing)Streaming mediaMappingStreaming mediaFactory (trading post)Uniform resource locatorComputer fontWindowCategory of beingServer (computing)Attribute grammarExpressionSystem programmingQuery languageDemo (music)Object-oriented programmingRevision controlMereologyState observerDiagram1 (number)OntologyVideo game consoleDifferent (Kate Ryan album)GeometryBitInformationPoint (geometry)Point cloudLink (knot theory)Pay televisionProxy serverSyntaxbaumNetwork topologyCuboidDatabaseTouchscreenType theorySerial portShared memoryMedical imagingOperator (mathematics)Social classStructured programmingNumberFunctional programmingPrimitive (album)Procedural programmingQuicksortMultiplication signPersonal digital assistantDirected graphWebsiteRight angleForm (programming)Parallel portArchaeological field surveyTraffic reportingString (computer science)Field (computer science)Online helpSequelNeuroinformatikComputer animation
51:27
Data modelSystem programmingFluid staticsContext awarenessExecution unitDemo (music)SynchronizationOvalStreaming mediaString (computer science)Function (mathematics)MathematicsDistanceSpacetimeLie groupPay televisionComplex (psychology)Point cloudQuery languageServer (computing)Virtual machineUniform resource locator2 (number)QuicksortDemo (music)State observerFunctional programmingWindowVelocityDataflowMessage passingGoodness of fitSerial portSyntaxbaumRootSign (mathematics)Point (geometry)Video game consoleExpressionLetterpress printingMultiplication signWell-formed formulaDistanceNumberNeuroinformatikBitStreaming mediaMathematicsString (computer science)Web 2.0SphereWindows RegistryEmailProjective planeCompilerSpacetimeProcess (computing)Division (mathematics)RadiusService (economics)Directed graphLevel (video gaming)Social classPlanningPhysical lawLocal ringSheaf (mathematics)CAN busClient (computing)Instance (computer science)Form (programming)Boss CorporationFigurate numberLine (geometry)Hamiltonian (quantum mechanics)Right angleKey (cryptography)Computer animation
57:26
Demo (music)SineMathematicsFunction (mathematics)SpacetimeDistanceSynchronizationFluid staticsEmpennageBinary fileContext awarenessSource codeProxy serverPort scannerSimulated annealingTask (computing)SummierbarkeitRegulärer Ausdruck <Textverarbeitung>Degree (graph theory)ExpressionExecution unitOperator (mathematics)Query languageContext awarenessPoint cloudDistanceNumberSurjective functionComputer fileDemo (music)Well-formed formulaProjective planeComputer programmingUniform resource locatorAbstractionState observerField extensionFluxAttribute grammarType theoryKeyboard shortcutQuicksortPersonal digital assistantStreaming mediaCodeFile systemLink (knot theory)MappingView (database)Macro (computer science)Degree (graph theory)Multiplication signServer (computing)Profil (magazine)MetreSystem callWritingRight angleSlide ruleFormal languageNeuroinformatikPower (physics)Functional programmingSource codeService (economics)Cellular automatonSystem programmingGroup actionPlanningProduct (business)Programming languageComputerLevel (video gaming)Complex (psychology)Process (computing)Address spaceSequelCode1 (number)Computer animation
01:01:47
Extension (kinesiology)Event horizonProcess (computing)SoftwarePrincipal idealSystem programmingLibrary (computing)AbstractionService (economics)Stack (abstract data type)Data modelQuery languageComponent-based software engineeringCore dumpOperator (mathematics)FeedbackService (economics)AbstractionOperator (mathematics)QuicksortQuery languageSerial portPoint (geometry)Projective planeState of matterRow (database)System programmingPlanningFeedbackOcean currentData modelExpressionLibrary (computing)Multiplication signEmailEndliche ModelltheorieSheaf (mathematics)MultilaterationWorkstation <Musikinstrument>Field (computer science)Computer animation
Transcript: English(auto-generated)
00:00
All right, good afternoon, everyone. Welcome to my talk about cloud scale reactive extensions. My name is Bart de Smet. I'm one of the original people on the RX team back five years ago or something. It's been a long time, more than five years now. And ever since I've actually moved a couple of years ago to the Bing team to work on large scale
00:21
event processing systems that power experiences such as Windows Phone Cortana, which is a pretty big scale solution. Today, just for Windows Phone, and I know that Windows Phone is not that popular in this continent just yet, we'll work on that. But soon it will be on every single Windows 10 machine. And so we're talking about millions and millions of users
00:43
that, as it turns out, all are actually writing indirectly, of course, RX subscriptions, RX queries in the cloud. And so I thought, well, it's 3 p.m. in the afternoon on a Friday evening, even though it never gets dark over here, it seems, so I can't really talk about evening.
01:01
But I was wondering, how many of you have been bored to death with PowerPoint slides and sort of say, please don't baffle me here with a whole bunch of PowerPoints? Please say yes because I wanna open up Visual Studio. Okay, good. So I thought, well, I have 40 slides here.
01:20
I will just upload them. You can go through them. They will contain a whole lot of background information. But instead of doing that, I thought, well, why don't we just jump straight in and sort of do some freestyling here, talk a bit about RX, the state of RX, the things we've been building, what we wanna do in the future, and what we've built in Bing, and what we're actually planning to ship
01:42
in the next couple of months to the next year. Maybe before we start, maybe a quick show of hands, how many of you have actually used RX? Okay, that makes it a lot easier for me. How many have been to either of the two maths talks? Like we have Matt over here who talked about message abstractions, and we had Matthew Podwysocki yesterday
02:01
or two days ago talking about RX for JavaScript. Like anyone has attended either of those two talks? Okay, good. So obviously the speaker was there. Yeah, that makes sense. Okay, so let's just jump in. And I actually created a little agenda just 10 minutes before we started over here.
02:22
With what I think is a good trajectory through this new set of things that we've been building. And so maybe I want to start a little bit by talking about the basic abstractions of RX because we wanna take those things to the cloud. We wanna do something fundamentally different than what we've been doing with RX for UIs
02:41
and in the web browser and all sorts of local applications. We wanna take it cloud scale. So it does make a lot of sense to talk about the basic abstractions. So given that everyone over here is kind of used to RX already, I think it will not be a difficult exercise to name me the essential interfaces.
03:00
Like who wants to have the first interface? I observable, yeah. So we have I observable of T. We have the other side of the equation which is I observer of T like this. Then we have a third one which is also in the BCL. What do you do if you smash those two things
03:21
together, what do you get back in RX? A subject, yeah, that's what you can do by smashing the two together. That's a nice subject of T. When I said smashing together, I also meant something like give an observer to an observable and you get something back. Like what's that kind of nuclear reaction give you? Subscriptions, yes,
03:40
and those happen to be I disposables. And then behind the scenes, we have a whole bunch of things that are the machinery, the heart of RX, like the thing where the concurrency happens. How do we call that thing? The scheduler, yep, the I scheduler. So those are kind of the fundamental concepts in RX. And just as a quick reminder here,
04:01
I'm just gonna give you the essential description of those guys. I observable has an I disposable subscribe, takes in an I observer of T, observer like that. This happens to be, is this an out or an in? It's out, yeah.
04:21
It's a co-variant interface, means an I observable of person, you can use to also observe things like students, like you can assign those two things. And then we have the observer as well, interface I observer, and that's the one that's in. You can give things in a contra-variant way here. And that actually has as the main method,
04:40
I would say on next T value. And then you also have on error and on completed, to deal with the end of the sequence. So those are kind of the essential abstractions. I disposable, very simple thing. Then you subscribe to an observable and you give it this callback interface called the observer, you get a disposable. And that's really at the heart of the composition of RX.
05:03
Because if you wanna create the query operator that says, well, I have two observables and I wanna merge them together for example. Well, we can subscribe internally to the left sequence and to the right sequence, both of them give us observables. And then we have to give you back an observable to say, when you are sick of getting those merged results,
05:23
we will have to do something to clean up. What do we do? We just merge those two observables or those two disposables together into one, and we can compose them and give out the composite disposable. So it's a very compositional way of doing things. So interface I disposable, everyone knows this one.
05:42
That's just for it to dispose. I'm a little OCD, there's a space missing here, good. And then we have subject, which is a very simple interface. Actually, there are two, T inner and T outer over here. And this one actually is a T observable of, well, it's not inner and outer really,
06:00
it's T input and T output. T in and T out, let me call it like that. Observable is the output, T out and it's an observer of T in, okay? So that's what we have. What does a subject allow you to do? Well, a subject you can think of as an addressable resource.
06:23
You can create a subject, just like you can create a list and link to objects. You can give that thing a name, a variable name, for example, and you can give it to somebody who wants to pump things in. And you can also give it to somebody who wants to listen to it. So it's kind of the in-memory pub subsystem if you want. Like, you know, it's the place where you push things and it's the place where you observe things as well.
06:43
So that's what we have today in Rx. Now, let's now switch gears and think about, well, how do we map this thing into the cloud? How do we build a distributed system using those abstractions that allows a phone over here,
07:01
I could have hooked up the phone to project it, but you know, when you go to Cortana over here and you say something like track flight BA49, how can that guy formulate a query that talks about a stream of flights, a subject of flights that's sitting somewhere in the cloud and tell the cloud, call me back whenever flight BA49 is late
07:22
and send me a notification on the phone. Well, I have something that needs to formulate a query over here. I need to put that query in the cloud. The cloud needs to communicate to my client at some point in the future. And of course, if I cancel my flight, my phone needs to say, well, that subscription that I created three months ago,
07:41
forget about it. I don't wanna know about BA49 flights anymore. So you need to be able to unsubscribe. So the problem becomes, well, if you have RX like this, of course you can write something like, you know, I subject of int, axis equals a new subject of int. That's my subject.
08:01
And you can do things like axis.subscribe and let me actually make this thing a little bit more realistic over here. Let me make it, well, first let us make it shorter, you know, by just firing a little bit. Say that it's a subject of flight and then it's not axis anymore, but Fs for flights. And I do something like where flight goes to flight.id equals BA49.
08:23
That's the one I always take. So that's the example we always use. But if you do this, well, you get back a disposable and over here, of course, you will put an observer that will receive the notifications for the flight. So you could put something here like new phone observer of flight or something. And then whatever, like those notifications
08:41
or something you come up with, like an observer that knows how to talk to the phone. But what's the rub here? What's the thing that doesn't work in a distributed system? It's on the left. Disposable is one of them. But more specifically, we are using something very neat here, something fundamental to the von Neumann machine.
09:03
Like I have memory addresses. I have freaking things here like called Fs and D. Those refers to my local working set. Like, you know, if my phone reboots, well, I will not be able to keep D around for three months to cancel my flight subscription. So somehow, we have a convenient solution here,
09:21
which I call intrinsic identifiers. The operating system or the CLI gives you intrinsic identifiers. Every object has an identifier and you can't even see it. Of course, if you do C++, you can see it. You can see the memory address. But that's not much use in a distributed system. So what we need to do,
09:40
and because we are X people, we always think about flipping things. Like, you know, those who've watched talks about the duality of enumerable and observable C. We like to flip things around to make the world a better place. Like observers become enumerators and those kinds of things. Well, here, I'm gonna flip the equal sign. I'm gonna say, well,
10:00
we're no longer gonna assign this whole thing to D. We're gonna give D to the cloud to say, dear cloud, I have a subscription. I want you to name it D. And then later I can come back to the cloud and say, remember that thing I called D three months ago? Please get rid of it. So we're just gonna flip those assignments around
10:21
in the fundamental abstractions and we create something called extrinsic identifiers better. We're gonna give an extrinsic ID to an artifact. Same with the subject over here and it's now pseudo syntax. You know, we're gonna do something like this. We're gonna say, I wanna create a new subject and I don't care about those local in-memory addresses
10:42
on my machine. I wanna give that thing an identifier in the cloud so I can go back to it. And more specifically, this allows the flight data provider, like British Airways and whatever flightstatus.com to discover my flights. To say, I wanna have the flight stream and I wanna publish a delay into it.
11:02
And somebody else says, I wanna subscribe to the flights. What's the name of the flight stream? And they can discover it and subscribe to it. So that's kind of the fundamental thing we have to do. Now there's a second thing, of course. What if this where clause in this FID equals BA 49?
11:22
Where's that gonna run? Not on my phone. I don't wanna get all the flights in the world streaming to me till my next flight and then filter locally over here. Supposedly if you still have battery left, like filter on your device, all the flights to find BA 49 for the day that you're leaving. Like that doesn't make sense.
11:42
So what we need to do as well, is we need to hoist this whole computation up into the cloud. And what can we do? What's the secret sauce in .NET to take up a computation and run it elsewhere? We've done it with link to SQL, it's called. Expression trees, yes, expression trees.
12:02
So we need to do something with expression trees. So I'm gonna ask you, how many think that this thing is a spelling mistake? Ike observable. I love the name. I invented it. Of course I love the thing. It's not a typo, it's a type that exists.
12:22
It's called Ike observable and I can pronounce it. It's beautiful. What's that? It's an observable spelled with a freaking Q. Now what can you do in LISP to catch a computation as a data structure? It's called quotation. This is a quoted observable.
12:40
An observable that has its representation of the computation sitting on its back. You can walk up to it and say, give me your expression. And then you can say, well, nice, that's an expression. What can I do with it? I could translate it to some Twitter language to filter things in Twitter. I could translate it to SQL, to send it to SQL database. Or I could take up the expression tree, serialize it
13:03
and install it in the cloud. And that's what all those Windows phones are doing. When you talk to Cortana, they formulate an RX query locally. We pick up the expression tree and we post it in the Bing cloud. And it's sitting there till your phone says I no longer need it. So that's what we do at large scale. We're talking here about millions of those things
13:22
sitting at any point in time in the cloud. Okay, so Ike observable. So let me actually show you one implementation of Ike observable where people said, that's a nice toy example. I wonder what they're gonna use this for. Well, I created an example a long time ago called link to WMI.
13:40
And WMI has something called WMI events where you can ask the operating system every time a process starts and the process is trojan.exe, it's a virus. I wanna get a notification. Well, you can do that using something called WQL. What's WQL? It's a bastardized SQL for the operating system
14:02
to query WMI Windows management infrastructure instrumentation better objects. So of course, when you say, there is some language called WQL and you talk to me, I say, does it have a link provider yet? And it turns out it didn't.
14:21
And it also turns out that it's events. So an Ike variable for an innumerable sequence does not make any freaking sense. Like you're not gonna enumerate the events, you're gonna observe them. So you need Ike observable. So link to WMI events is over here. Here's a very simple example where we say,
14:40
again, not with a spelling mistake, dear event provider, I wanna observe the process star traces, not a joke. It's shipping code. And this process star trace, of course, you know, every time you do something like this, you need to have some object mapping and you can put some attributes on it
15:00
to say process star trace has an internal name, which is weird and I wanna make it nicer. So that's what I'm doing here. This is a so-called MOF definition. Not gonna make a joke about Germans here. It would only work for people from Belgium who call the Germans the moffen at some point. Like whatever, like the MOF definition sounds like weird to me at least.
15:23
But so that's the internal name. It has a couple of attributes on it. And now I can say if there were any Germans in the room, please still press the green button at the end. You know, like I don't wanna have any issues here. But so what I do here is I observe the process star traces and I do a where and a select.
15:43
That's just standard link. But the way it's gonna execute is by giving that thing as a WQL query at runtime to system dot management of instrumentation to ask Windows, please do that for me. And of course, it's a lot of electrons go in two ways
16:00
because the C sharp compiler takes your code, turns it into an abstract syntax tree then generates code that at runtime will create an abstract syntax tree that then we reduce into WQL, which goes into a WQL parser in the operating system, which creates an abstract syntax tree by the way, and then executes the thing. So like there's a lot of wasted energy here, like you know, in those kinds of systems,
16:21
but that's what we do to marry those things together of course. So when I execute this thing, control F5, I'm running this thing as admin. You see that my link query translated into this thing that really looks like SQL, but it's an event query in the operating system. And so I can translate an expression tree
16:41
from C sharp into this domain specific language. That's an IKEA observable. I did not see at any very point in time that this thing is gonna run elsewhere. I can write a where clause, I can write a select, I can knock myself out here and use anything.
17:02
Of course, it may throw at runtime as with any link provider, as Matt also pointed out some time ago. It can always feel at runtime saying, well, you're trying to do an order by here, but this database doesn't support an order by, here's an exception for you. In fact, with Roslyn, with code analyzers and things, we could do that at compile time as well now.
17:21
We could actually look at this thing and say, we're trying to write a link query against the observe of an event provider. Let me provide some squiggles for things that will not work at runtime. You can still compile it and it will compile and at runtime it will complain, but Roslyn will say, well, I told you so. This will not work. So now you can do that if you really want to.
17:42
We haven't done that yet, but it's definitely a possibility. And here's the other thing. That's like a diode or a membrane that only goes in one way. As observable is something that says, well, just like as enumerable in link to SQL,
18:00
if there is something the server can't do, we are still here to help you out. Just put this as observable in and that's kind of a barrier that says everything to the left runs remotely, everything to the right runs locally. So that basically just changes the type from observable, which uses expression trees to observable, which is just in memory.
18:22
So that allows me to sort of do this. So that's cool. Now, does that mean that if I want to submit this thing, I have to invent a different language? Like this thing is WQL, the language already exists. And that's why link was so successful in some senses.
18:40
There were a shitload of languages out there that deal with the numerable sequences. Like SQL and things for XML and entity framework and blah, blah, blah, and a whole bunch of like foreign things, like even query strings in the URI for Twitter can allow you to do a simple query to get the latest tweets for a particular topic. But in the event space,
19:02
there's not really any languages to say, I want to have a query that describes filtering of events or something. So we could do two things. We could say, let's invent a new language, or we could say, well, why don't we just try to pick up this expression tree and just transplant it from one process to another? And that's what we do in Bing.
19:21
So that's one of the things we'll actually ship. And then I'll get back to the observable abstractions. I'm sort of showing you an implementation detail now. This is called bonsai serialization. And we're gonna ship this later this year. And bonsai trees, well, it's a cute name I came up with because they're kind of compact. So I said like, why don't we call it bonsai trees?
19:41
You will see that they're not that compact if they're in verbose mode, then they're more like oak trees, you know? But so, bonsai trees are kind of an intermediate format that allows to take a .NET expression, translate it to a JSON format, but we also have a binary format, which is super compact. And you can also send it on the other side
20:00
to an expression tree. You can serialize and deserialize. But notice that in this diagram, I've inserted expression slim, some intermediate state. And what this allows us to do is actually have expressions that have no type information. And this allows us to come from TypeScript, for example, to Python, from Python to .NET,
20:20
from .NET to Python, all those kinds of combinations. As long as a language can capture an intent of a query as a bonsai tree, we can serialize it and bind it on the server to an implementation. In Bing, it happens to be C++ on the phone to .NET in the cloud. That's what the bonsai tree serialization looks like for us.
20:41
We don't have .NET to a C++ runtime yet. We could do that, but we don't. So you can sort of see this thing becoming a pretty interesting language to sort of translate things into. Let me just do a very small example. Let's say I have an expression.constant 42.
21:02
Always needs to be 42. And I will create a new bonsai expression serializer over here. And this thing takes in two weird things called lift and reduce factories. I'll sort of skip over it a little bit. I'll just do something magical here. Ignore this for just a second.
21:20
I will explain it in just a second. I've run out of variables already. Like this, okay. What this really is is like lift and reduce just says if the expression contains an object, in this case, 42, how do I turn this 42 into JSON and how do I turn it back?
21:41
And those lift and reduce factories take in the object and the type and on the other side, the JSON and the type. So we can sort of plug in any other serializer that already supports serializing things in a type-aware fashion. In reality, we plug in Newton's of JSON. If we do it in textual format, if we do it in a non-textual format,
22:01
we plug in Microsoft Bond, which I recommend you to take a look at. We open source it on GitHub and that's the serialization mechanism that we've been using inside Microsoft for all the data in Bing for a whole bunch of years now. It's written in Haskell. We have code written in Haskell. It's super performant, super compact. In fact, it's faster in a number of cases
22:22
than protocol buffers from Google, just pointing it out. So Microsoft Bond, all open source, they can use it from .NET, C++ and a couple of other languages that also have bindings and the community is totally free to implement bindings to other languages if you want to do so.
22:40
So that's the serializer and what I can do with the serializer is I can serialize an expression. But take a look here. It takes in an expression slim, not an expression. How do I create a slim expression? I just do two expressions slim on expression. So I can sort of go from a .NET expression, make it slim, which sort of makes type information optional just so happens that we keep
23:01
all the type information in this case because if you have some diamonds in your shelf, you don't throw them out to say like, let me satisfy this constraint. Like you keep your diamonds, right? So if you come from a typed language, it would be a waste to throw it away. And then you get out some JSON over here. And I could actually pretty print as JSON
23:20
using an internal JSON parser that we have, which also creates an AST. Like most of those JSON things don't really create a boot AST. So we have to create this thing at some point, but this allows me to do Microsoft.json.expressions.prettyprinter, create a new pretty printer and visit the object,
23:43
the JSON thing. And then I will get a string back, which will just be pretty printer JSON, just with indentation as opposed to one long string. So this is just my quick way of doing pretty printing for JSON here. But what this will give me, oh, damn it. Like, I already had a T somewhere.
24:03
The problem is I can't do demos that contain more than 26 local variables. Then I run out of space. But so this is what the bonsai tree looks like. And it's not that bonsai because it comes from .NET At the top, it contains a whole bunch of type information. That's called the reflection context.
24:20
It contains all the information about the types in my expression tree. But the tree itself, if you would come from JavaScript or something, would just be colon, which means this is a constant, and then the value 42. This zero is a reference to system 32 in the table above. So we can sort of retain the type information, but that's optional. That can sort of be taken out and then we can figure it out at runtime.
24:42
So I can now also do a bigger expression. Let me do an expression like the one we saw before. I could do an expression of func from int to bool that takes in an X and then X percent two is equal to zero. So for even numbers, I don't have an X yet.
25:01
Oh, I have an X somewhere else. Ah, god damn it. Y. Don't have Y, right? Ah, I have Y. Z. Why don't we have a larger alphabet just for developers? Okay, that would be kind of cool. But so that's actually a lambda expression being serialized over here.
25:21
So it looks a little big, but you see the arrow, that's the lambda arrow. You see the equals, the modulo, that's the percent over here. You see pretty much all the information that you have, like you see the constant two, you see the constant zero that you're comparing to, and you see the variable named Z, which is my parameter of my lambda. Z goes to Z as modulo two equals zero. So we now have a serialization format
25:42
that allows us to take an expression tree from place A to place B. So that's what we're gonna do in the cloud. So now let me actually switch to the Bing stack. And I have the Bing stack running over here in what we call a one box solution. So just a local deployment kind of thing.
26:03
And I have a client that pretty much shows the code that phones see to interact with our system, the data publishers see to interact with our system, and also have a service host over here. The service host is arguably the most complex thing, and the thing I will spend the least amount of time on,
26:21
but the way you think about this is, it's the place where you bootstrap the server. And we do that, of course, during deployments. We'll say, please deploy this to 100 nodes in the cluster and execute this piece of code and start listening on some sockets and connect to a data store and do this, that, and the other thing. And that's what we call the reactive platform.
26:41
The one thing I wanna show you for the reactive platform is you go to the platform over here. The platform has a bunch of roles and there are two essential roles in our system called the query coordinator or the QC and the query evaluator, plural, multiple query evaluators. That's because it's a local box, we only have one QC,
27:00
but you have multiple query evaluators. What do query evaluators do? The word says it itself. Those are the things that contain all those expression trees compiled and bound to our X. So like when you upload your phone, a query into the cloud, it will go to the query coordinator, it will look at your query and say, that's somebody who wants to do flights,
27:20
he wants to do something quite expensive. I have a machine that's available, let me put this query on query evaluator 52 and it's sitting there and it gives a mapping table and a dictionary so we can find it again, all of that kind of stuff. The query evaluator then is a stateful service that contains the state of those RX queries until you suspend them or delete them.
27:42
So that's what the query evaluators do and they actually run an implementation of RX. So by the way, the way that those query evaluators run is this customer story here, I opened it before the talk. This is running on Microsoft Azure Service Fabric. That's a new technology we shipped at Bing,
28:02
not Bing, always talk about Bing here. That's the thing we shipped at Build this year. It's a technology in Azure that also runs in our data centers that allows you to build highly available distributed services. And it's actually a technology that you can compare to a combination of key value store databases,
28:20
leader election systems, a lot of zookeeper, a whole bunch of things kind of in one big technology. And you can do actors with that. There was a talk here a couple of days ago or yesterday about doing actors and microservices using the Azure Service Fabric. We use it for the highly availability of those queries. Like when you upload a Cortana query,
28:40
it's sitting in three different data centers. In case one data center goes down, we can fill it over to a different data center. And that's what Azure Service Fabric allows us to do. And here you can also find the code name of the project. The project is called Reactor. Reactor is the system. Unfortunately, there's some other technology
29:00
that also calls Reactor. That's what happens when you sort of all dabble in the point of words that start with react and there are only that much. In fact, there's only 26 letters as we came to conclude just a moment ago. So there's not that many combinations. So Reactor is basically the technology that I'm showing you here right now.
29:21
And you can think about the foundations of Reactor to become RXV next. So let me now switch to the client side here. And before I do so, let me actually boot up the server. Otherwise, we're not gonna see too much. And this thing is running the query coordinator, the query evaluator, storage, messaging,
29:40
the whole shebang in one console window. I could make it more impressive by basically saying, spawn a separate window for everything. But then we run out of memory on the taskbar in Windows 10. That's not true, by the way. But there would be a lot of windows all over the place and I don't really wanna do that, okay? I could actually put them on a separate virtual desktop, which is new Windows 10 feature.
30:03
Try to be a little salesman here. Okay, that didn't work. Okay, so let me go back to the demo here. And in the demo, well, let me first show what the demo looks like. The demo is kind of a canned demo that sort of has something that's guaranteed to work
30:22
because if you freestyle, it can always go wrong. So I have in here in this menu, one to 16 of rescue lines. If something goes wrong, I can call those things. 99 is the freestyle, which is an empty project kind of thing, an empty file. And we're just gonna play there. In case something terribly goes wrong,
30:41
I will cry under the desk and then I will press a number between one and 16, okay? So hopefully it will be 99 all the time, you know? Okay, so let's go to demo 99 and be brave. Okay, good. Who knows link to SQL? Good. You can kind of leave because what I'm gonna show you
31:02
is an API that's isomorphic to what you already know in link to SQL. It's exactly the same principles, a little bit extended, okay? What do you do in link to SQL? Where do you start to write link to SQL queries? It's this magical thing called the context,
31:21
the data context, exactly. So what are we gonna create? We're gonna create a context, good. We're gonna create a new context, client context, like this, and the client context takes in, ah, link to SQL, it takes in this connection. Doesn't take in a database connection, of course it takes in a reactive connection. That's the physical layer, logical on top of physical.
31:43
So I'm gonna create a connection which is gonna be, I will just cry out for help here because I want to copy this complicated type name. Over here, remoting service connection. And that thing takes in a link to SQL, local host port 1433 or whatever, right?
32:02
In this case, it takes in the URI of the server, which happens to be in this case running over, guess what, you know, I'm a hardcore guy and I don't care about deprecated technologies. This one box solution uses .net remoting. Like, you know, no WCF and stuff for me,
32:20
like, you know, just hardcore proxies, you know? Keep it simple, you know? Most people on the team did not agree that remoting is simple after a while. Okay, fine. So that's what it is. So now I have a connection and a context. I can forget about the connection. That's actually where you do all the dirty stuff like timeouts, authentication, blah, blah, blah. Like things you never want to see in your life.
32:41
But here in context, you're in the unicorns and rainbows world where you have very nice abstractions. And the abstractions you have are not get table, of course, like, are you crazy? Events are not stored unless you have an event stored. Events are just flowing. So you have observables, observers, and all those things. The world is beautiful
33:01
because we already have those things in Notepad. We have observable, observer, disposable subject. Like, so you will find getters for all of those things. So let me now go back to Notepad here and try to write this line. And I'll simplify flight for a moment with an int. Just not to have to deal with types for a second.
33:21
But how do I create a subject that can send and receive ints or flights and give it some identifier? Well, you have different kinds of subject in Rx. You have subjects, you have replay subjects, you have behavior subjects, all sorts of different ways you can route messages and maybe have memory.
33:41
So you need to have a new concept that sort of encapsulates how do you talk about, I wanna create a subject and not a behavior subject. We call that thing a subject factory. And actually we give it a nickname. In the API, we call it a stream factory because people like to talk about streams. They think that subject is a mail system
34:00
and then they look for the to and the from as well and that's not there, right? So not subject, stream. So I'm gonna create or get a stream factory and I'm gonna say, I wanna have a stream that has a particular input type and a particular output type. Remember that subject in Rx
34:22
has exactly that signature, input and output, okay? And now I have to say what kind of stream factory and this is where my story about the extrinsic identifiers comes into play. Every single artifact that existed in Rx and that I wrote down in Notepad
34:41
that I could give an instance for and create a local variable name for, all of those things have identifiers in our system that are based on URIs. Why URIs and not just strings? Because some people like to think about stuff like semantic wrap and blah, blah, blah and they like to have a hierarchy of things but we treat it as an opaque identifier.
35:00
We will not split it based on slashes like it's still a string to us. But so it's a URI just to bike chat the whole world, right? So I have to pass in the URI that's for the stream factory that can create simple streams and we call that thing the firehose stream factory. That's just a local variable name
35:20
which is just an alias for something reactor platform bing.com slash blah, blah, blah. Like some URI that identifies simple subjects and of course we have them for all the other things as well, okay? So I can say I want one of those stream factories and now I have a factory. Now, just like link to SQL,
35:41
at this point we have not gone to the server. We just have a local in-memory proxy that contains the potential energy that we can turn into kinetic energy when we go to the server. Like it contains just the intent, right? Now, what's the conversion from potential to kinetic energy? In link to SQL, it's simple.
36:03
You do for each and poof, things start happening, right? For each can be blocking. It can take a long time because SQL takes some time to respond. Our world needs to be async. That's why this thing is run async. So whenever you see something like create async, you take the potential energy on the left which is your expression of what you wanna do
36:23
and you send it to the server. The async is kind of a religion here. Every time you see async, you leave the client, go to the server. Like we sort of are very strict about that. So now I'm going to the server and I'm gonna say I wanna create a new stream and I need to give it an identifier. So what's this firehose stream factory?
36:41
Let me highlight it in Notepad here. The firehose stream factory is this part. It's the replacement for a type name. I'm saying I want a subject, not something else. What's the identifier I have to pass in now is how I want to name my new stream that I'm creating using this factory.
37:01
In the regular factory, you do create and assign to a local variable but the variable now has to be passed in. You have to give it an identifier. So I'm just going to go here and I'm gonna say like create me a new stream and let me call it my demo slash access and I will await that thing. This is kind of the equivalent of create table
37:22
and then access in SQL. And if I don't want a table but I want something else like a view, then I would use a different factory over here. Like, you know, so this keyword here, table or view is basically encoded using that URI, okay? So of course we're not creating a table,
37:42
we're creating a stream. What this does is it goes to the QC. The QC says there's a new stream of ints. I need to configure the messaging role to start accepting those things, set up connections, blah, blah, blah. Underneath the hood we use all sorts of communication protocols, CMQ, blah, blah, like, you know, whatever you have,
38:01
all sorts of transport mechanisms. So this thing will configure a whole bunch of things that allows messages to flow into the system, okay? So now I have a stream and that worked. Now let's assume that somebody wants to subscribe to that thing. Like my phone says I'm really interested in even numbers.
38:20
Okay, nice. How do I write a query for even numbers and put it up in the cloud? Well, what I'm gonna do is I'm gonna say context getObservable, the thing I wanna subscribe to, it's an observable of int, what's its identifier? I just gave it to the subject. I said, I wanna call this thing access.
38:41
So now I can pass in this URI. So it's decoupled, like this doesn't have to be the same app, but this can be a totally different app and you can tell somebody you have this thing. By the way, on the context, you also have metadata. The context contains three things. It contains proxies, which is like DML statements, right?
39:01
Like, you know, you sort of do select queries and that kind of stuff. It contains a whole bunch of define capabilities to define things that's like DDL, data definition language, where you say, I wanna create a stored procedure like a macro for something. Third thing it contains is metadata. I can write queryable queries over the observables collection.
39:20
That's what we do to create tools in Server Explorer. That basically you connect to a reactor and it shows you those are all the streams you have. Those are all the observables. It's like the SQL catalog in a database, right? Like, so we have those three concepts here. So you can discover and you can create, okay? So I will get the proxy to the access observable.
39:40
I will also get an observer called the console out observer. It's an observer of int. Where does that one come from? Well, my server deploys that one by default, like, you know, just to do some printout debugging. So here you go. Console observer is reactor platform bing.com slash observer slash console. Okay, so that's the identifier for that guy.
40:02
Just has a name. As long as I know this thing, I can refer to it. So now I have access and cout and I can do access dot. Look, that's a familiar world. Even though I'm talking about the cloud here, I see all those beautiful RX operators. It's a subset of them at this point,
40:21
because those are the ones we need in the cloud. But we can write a where clause and we can say X goes to X larger than zero or X percent two equals zero. That's the thing we can see like using bonsai trees like we saw before. And I can do a subscribe. Oh, I can't do a subscribe. I have to do subscribe async.
40:40
Of course, because that's the place where we leave the client, pick up the computation and put it in the cloud. So it has to be async. And by the way, this thing is used in different places. The phone talks to the cloud to upload queries for things like flight status, but for things like time to leave reminders, which is you have to leave now
41:00
if you want to get in the bar at time, such and such, like then we give you like a plan and then you have to leave. For that, the cloud ships a query to the client to monitor your GPS location, to know where you are half an hour ahead of your meeting, to know like where to send you to, like and to remind you if you're already there,
41:20
we don't have to send you a reminder. So we ship those queries from A to B, from B to A. So this thing has a client context sticking out. And this thing in the cloud here, let's say that's the cloud, also has a client context sticking out so they can talk to each other. And this allows for an interoperability system. Like, you know, you can actually have a query you send to one system. And that system says, I can't do that part,
41:42
but I know somebody else who can. So it can take part of the expression tree and say that thing goes over here. And now that thing goes over there. And the whole thing is called this. When you dispose it, I go there to dispose that part. I go there to dispose that part. So you can decompose the computation.
42:01
So I'll do subscribe async. Of course I pass in the observer, but I also pass in a URI that identifies my subscription so that I can come back to this a couple of months later to say, get rid of it. So I will pass in a new URI and I will call this my demo sub and I will await this thing.
42:21
This is what the code on the phone could do to say, notify me whenever there's an even number, which is very useful. So let me do control F5 on this one. Again, hit 99. Here we go. Ah, and now my cloud, this little box here, which is sitting there that silent because it needs to do millions of those things. So like, you know, it doesn't want to print
42:42
like tons of messages here. I could configure some logging and you would see things happen. But this has sent a Bonsai tree from the client into the cloud, into the query coordinator. And that one has chosen a query evaluator, which is all sitting in the same console window here and has compiled an executed expression over there. And it's now persisted. If a node goes down, the service fabric will replace it
43:03
on a different machine and bring back the computation. Okay. So it's sitting there now. I could do 99 again. And you see that the query coordinator now says, you already have this thing. Like there's a name conflict here. So that's what you expect, right? Like if you create a table on top of a table with the same name or whatever, you will get this kind of right conflict.
43:22
So that's the message that you see over here. So that's expected. That's not a call for me to go under the desk and cry. This thing has not worked. This was expected. Okay. Good. So now we have a subscription standing. What's the remainder part? Now somebody has to publish events into the system.
43:40
So how do we publish events into the system? Well, some other party can walk up to the same cloud and say, give me a proxy to the observer side of my subject. So I can publish things into you. And I can say, give me the observer for, let me copy that thing. Again, for demo access, which is my stream, right?
44:01
You know, the subject where I push things into and now context dot, you can guess it on next. We leave the client. So it's async, on next async. And you can pass into here a value, say 42, and you can await that. So now flightstatus.com can await PA delay, await this and sort of like send all those things.
44:21
The await allows for the acknowledgement to come back and behind the scenes, there's a whole bunch of other things happening like sequence IDs and whatever to allow for like deduping and all of that kind of stuff. But so this is what happens here. So let me do control F5 on this one. And this will send a message into the cloud and print it to the console.
44:43
It doesn't really print to the console. It's of course the server console because the whole expression tree that I created in the previous step has been uploaded to a query evaluated in the cloud. So if I would do this for real in Bing, I would now have to drive up to three data centers to look on all the console windows to see where the query was running. We have some more sophisticated tools as well,
45:02
which allow you to know where things are running, obviously. So you see it comes out on the console. Of course, in reality, it doesn't use the console observer. In reality, it uses the windows notification service observer. So it can push 42 to your phone and give you a tile 42 has arrived, right? Like, you know, don't use the console,
45:20
use something else. So now you can do all of those beautiful kind of queries. So let me not dig under the desk here, but actually just show you the other demos I have here to give you a sense for the kind of queries that Cortana can write. What I'm gonna do in step one, and I'm just gonna step through all those steps
45:41
after showing you the code, as opposed to going back and forth. Actually, let me just open it on the side here, maybe. Oh, look, new Windows 10 feature. That's why I did that. Okay, nice. So demo one, it's not that useful, actually, in this case, because my font is huge.
46:01
But demo one is going to get the stream factory and it's gonna create a stream of geo locations. The only difference here is that I'm not creating a stream of ins, but I'm creating a stream of those locations. And what we do, because we obey to some ontologies for describing the data, we actually put attributes on those things called mapping attributes. Very similar to column attributes
46:21
and link to SQL or whatever. Like, there's so many mapping technologies out there, so we have to do something else. Just whatever. So, mapping attributes. So this allows me to create a stream for the BartDE location. So that's my location stream. That could be my phone and says like, hey, I'm Cortana.
46:40
Can I help you out with giving you traffic information? And you say yes, and it says, ah, well, then I need a stream for your location when I need to subscribe to you. I need to have you publish your location into the cloud so we can do some computations. So that's like Cortana doing that at some point. So now we have a stream for my location, which we can subscribe to when we need to.
47:01
And what I'm gonna do in demo two is I'm gonna simply subscribe to the location over here with the console observer. And what I'm gonna do, and let me actually try to get this a little smaller over here. Okay, like that. Well, it doesn't look that good now. But I'm just gonna subscribe here.
47:21
Notice one thing, we don't share types at all. That's one thing that the Bonsai trees allow us to do. Remember the expression slim thing? We have expression and expression slim. Expression slim allows to get rid of all the CLR type system stuff and sort of erase all the information for structural types. Guess what? TypeScript has structural types too.
47:41
Fits hand in glove. So what we do here is we just have a class that's actually spelled in Dutch in this case, because I'm a Dutch Cortana. And you know, the Dutch Cortana here wants to subscribe with the console and it doesn't have to share a .NET type. It just needs to know about those attributes. Standard stuff, right? So two, I'm creating a subscription
48:02
to the Barty location that will come to the console. What's number three? Number three is some other person sending the location. That's my phone uploading its location into the cloud and somebody there is subscribed to my location. Guess what? In this case, I use location yet a different entity.
48:21
Like, you know, it's defined over here, same mapping attributes. So the server knows what I'm talking about, but it doesn't care about what property names you give it. You can omit properties you don't need. You can add new properties, whatever. It's kind of completely structurally typed. So I can now do three and three will actually publish locations, those locations, which are actually actual locations
48:42
of a walk I recorded at some point. And it's publishing those into the cloud and my subscription is sitting there listening to them and it's printing them to the console. So what's demo four doing? Demo four is the demo that gets rid of a subscription.
49:01
I haven't talked about that one yet. How do you call a dispose in the cloud? Now you get a proxy to the subscription. You say get subscription and what do you get back in async disposable? In reality, it's something a little bit more funky. I will spell out the name that we came up with for all of those things. Like when you do get observable, what you get back is an I async
49:24
reactive observable of T. We will get rid of the reactive part in the middle there because we didn't want to have name conflicts with the future RX 3.0. It will likely be I async observable of T. I async observer of T. I async object of T. I async subscription of T.
49:43
And of course you also have the ones with S which are the ones that run on the server that don't use expression trees. Like we have those parallel worlds. Expression, no expression. Async, not async. Like it's a very symmetric diagram. So those cues look like madness but they totally make sense.
50:01
Like they just mirror images of the other types. So in this case, I get back an I async reactive subscription which I can dispose. And so this dispose async here is going to go to the cloud, serialize this expression tree. The expression tree is just something that contains the name of my subscription and it will dispose it. So when I do four,
50:21
I've gone to the server to dispose it. And when I publish back into the bar T location stream, you see my server is no longer printing stuff because somebody disposed that subscription. Now we can do much more interesting queries than just subscribing and publishing of course. So let me just zoom out a little here or make the screen a little bigger
50:40
and show me a few other things first. One thing you can do is you can define new artifacts as well. Let me reveal you an interesting secret here. Unlike most database systems, we have no built-in operators. We ship empty boxes in Bing. Our system doesn't know about where,
51:02
it doesn't know about select, it doesn't know about anything. All of those things have been defined during the startup of the server as just anything else can be defined in the system. Like this really means that stored procedures and all the built-in things are at the same footage in our system. Like we have no primitives built in, they all get deployed.
51:21
So in this case, I want to deploy an HTTP observer. We don't have built-in functionality to do HTTP. What we do is we say we want to define an observer that's parameterized on a URI where I want to send things to, that's my end point, and it's an observer of string. You can give it HTTP bodies.
51:42
In reality, it's not string. In reality, it's an HTTP request which contains headers and a whole bunch of things I don't want to know about. But what it takes in here is the URI for that HTTP observer so that other people can discover it and it then contains an expression tree that says how to create one of those guys.
52:00
And so we deployed that HTTP observer assembly in the cloud. It goes into some registry where we can download it from. And whenever somebody uses that thing, we stitch in this expression tree into the user's expression to say, you want an HTTP observer? That's how we're gonna create it. So we're basically just a big compiler in the cloud that smashes together all kinds of expression fragments
52:22
and then poof, instantiates it. Of course, in a rather efficient way, because we have millions of those guys sitting there. So this is gonna define an HTTP observer in the cloud. So I'm gonna do five now. And of course, what I'm gonna do once I have an HTTP observer,
52:40
you can guess it in demo six, I'm gonna create a subscription that actually sets up a web server locally. And then as opposed to subscribing the location to the console out observer, it's gonna get the location stream here. It's gonna get the observer I just defined and parameterize it with the end point I wanna talk to.
53:04
And then it's just gonna subscribe the bar to your location. It's gonna project it into a string and it's gonna subscribe it to the HTTP observer. That means messages that come into the stream get projected onto an HTTP body, flow into the HTTP observer and that one does an HTTP post to me.
53:22
It does no longer do console.write line, it does HTTP post, right? So that's what's happening here. And I'm running the server over here myself. This is getting a little closer to what we do in Cortana. In Cortana, we pass in something called the channel URI of your Windows notification service. And then we get the locations back streaming to you.
53:43
So when I do six, I have a subscription now and when I publish into the bar to your location, you see now the server in the cloud is posting things back to the HTTP observer to me. So now I have a subscription that does that. So that's cool. I can get rid of that one with seven that's just dispose async on the subscription.
54:03
I will skip over eight, but I will show you nine to show that we can do really complex queries. And in fact, there are some really complex queries in Cortana. This is one which is close to something we actually do. This is the mode of transportation query that given your location,
54:21
will determine whether you're standing still, walking, running, or sitting in a car. We also wanna do planes, but turns out that you don't turn on your phone on a plane, at least you shouldn't. So what you can see over here is a pretty complex query and I could explain it, but we all know RX. So I'm just gonna sort of give you the gist of it.
54:43
What you do is for every two points that you get, compute the distance in time and the distance in space. How do you compute the distance in space between two coordinates? You go to Wikipedia and you look it up. And that thing is called the have a sign function. So this query over here contains
55:01
this freaking long formula over here. Two times R times the arc sign of the square root of the power of blah, blah, blah. That's just copy paste Wikipedia, right? But you put it in there and we can take that thing from the client, serialize it to a bonsai tree and run it in the server. No sweat at all. Like any C sharp expression we can pick up, right?
55:21
Like, you know, of course you will have to pay for it in some way because it takes a little bit of compute. But here you see like high school mathematics 3.6 that looks like something 60 times 60 number of seconds per blah, blah, blah, per hour or something. And then some division here, that's a velocity. So this thing will calculate the velocity
55:41
and this magic number here is the radius of the earth which the have a sign function needs and assumes that the earth is actually a sphere. It turns out that it's kind of spherical in places where Windows Phone is popular. So the other places where there are big mountains, nobody uses a Windows Phone. So it's fine. It sort of flattens out. So yeah, Mount Everest like don't have reception anyway.
56:04
So it's good enough. So I can subscribe to the mode of transportation subscription. I've done nine here. And what I will do now is I will actually publish into Barty location. And then over here you see my velocity in kilometers per hour. The reason they're kind of ridiculously high
56:22
is because that's my walk from home to work which takes 10 minutes and now sped up into 10 seconds. So you have to divide everything by 60 which is a reasonable walking speed. Actually the 256 here and the 258 are very actual situations on my walk from home to work namely a traffic light.
56:40
So that's actually actual data that I recorded using the phone upload into the cloud and just let my machine print it out and then I use it for a demo. So that's the mode of transportation subscription. I can actually dispose that one. Oh, I think I just killed my server. Oh yeah, I pressed any key to exit.
57:01
Ah, stupid me. I'll just restart it. That's fine. Starting the Bing cloud takes a little longer of course. So I'm not gonna dispose it anymore because I didn't have persistence configured here. So I don't have to dispose it. Whenever this process dies, it sort of like just gets rid of everything. But what I'm gonna do one more thing here to finish up
57:23
is I'm gonna show you two more things. First of all, I'm gonna show you that this thing contains a formula that I don't wanna write every single time again. What do you do in say any programming language? Doesn't even have to be SQL when you're sick and tired
57:41
of writing the same thing over and over again. You create a user-defined function, right? A location in degrees to a location in radiance. Let me create a macro for that. Okay, so what I can do is I can, let me go here, define custom observable number 11. I can go up to my server and define the observable
58:05
which is parameterized on observable of location and will return an observable of location. I call it to radiance and it contains a projection of degrees onto radiance. And guess what? This thing here with this URI, something very powerful.
58:24
If I go to the select operator and I have 12 of that guy, you see that the select operator also has a mysterious attribute on top of it. I can now create an extension method called to radiance with that attribute on it. Just use it in my query
58:40
and it will bind to that thing I created on the server. So I can create little DSLs that do things like bar the location dot to radiance dot compute distance and all those kinds of things. So they are just expanded into those macros that I've defined before. And we are honest. In fact, select is mapped onto this thing. So very simple, consistent view of the world.
59:01
No more, you know, mythical names or mythical methods that are treated specially. Now with this, I can actually do one more example here, which is 15. And 15 does all of this mapping stuff that we talked about just a second ago. And this is not ask Mike,
59:21
this is ask Homer about the demo. What this one is, is a query that uses this to radiance that I defined just a second ago. And this is completely the same as link to SQL. In there, if you define a strike procedure, you create a dot net method, you put an attribute on it and you call it quits.
59:40
Like, you know, you can just map that thing. So here I've sort of gone totally crazy. I created my own context, which can be totally cogent because we have all the metadata, remember? Like you can walk up to the context and say, give me all the observables, give me all the observers. And in fact, this whole context, all those extension methods are cogent. We just walk up to the bank.
01:00:00
and say please generate me a context for all of those things of course if you ask it generate me a context for all the streams then we get the C sharp file that contains all the user streams in it and it doesn't fit the file system so like you know you can't do that kind of thing because it's really huge but like in this case I've sort of said I would like the two radians
01:00:20
operator to be brought into scope for me so I can just type two radians as opposed to all of this get observable get observer all of that stuff I want to have the Homer location and I want to have the Bart location so it generates code that has a known resource attribute and sort of wraps those calls for you and now I can write a very simple query over here that says
01:00:45
the Bart location converted to radians combine it with the latest location of Homer so whenever either of the two people are moving you know we're sort of gonna combine their locations to compute the distance between the two and the very first time in the future that the distance
01:01:03
between those two people is less than 100 meters then you will pop up a notification on Bart's phone to say Homer is nearby you have to ask him something or if it's the opposite thing that could be a strangle reminder for Homer to strangle Bart when he comes home right you know he's within 100 meters a little boy that kind of thing and so I'm kind of used to
01:01:24
those things because all my profile pictures are Bart Simpson's you know so this shows kind of the power of all of this mapping stuff right where you can sort of take all those abstractions you can sort of deploy all those reactive artifacts into the cloud and then sort of write computations and upload them using very simple Rx language so what's next for us I'll
01:01:47
actually not show you the 40 slides now I'll just go to the last slide what we are gonna do and this is the current plan of record there may be a couple of little adjustments on the way of course but we have a pretty big system it's took us about two and a half to three years
01:02:02
to sort of come from standard tracks and build out this infrastructure if you know pretty substantial amount of people at some point so what you see over here is kind of how our system is layered you have the query operators you have something called a query engine you have a data model which is the thing that allows for serialization deserialization you have a whole
01:02:23
bunch of service abstractions around it which use the service fabric as your service fabric and so what we're going to do is we're actually peeling the onion of the system and the goal is to sort of ship things inwards to outwards right so the operator library that supports new things like saving state and loading state it's first thing that will come
01:02:42
out the query engine would be the next thing things like expression serialization all those prerequisites to enable this kind of capability those are all kind of auxiliary things which will come out independently but before we need them in those projects and the goal is to put all of that stuff eventually at least up to the data model layer potentially up to the
01:03:04
service layer because now service fabric is publicly available so we can actually do a lot of stuff in the open there in the past we had this technology only internally but now it's publicly available but eventually all the way up to the to the top we would like to to get the basics on there so you can actually deploy this run this give feedback and all of
01:03:25
that stuff so what I would like is feedback if you think this is cool let us know if you think there's something missing or you want some clarification let us know the only emails that I don't like to receive is why is observable spelled with a Q you should rename it like you
01:03:42
know those are kind of the bike shedding syntactic questions I like the semantic questions first and later we can address the naming potentially translate the whole API to Dutch which will be much more understandable than observable and like you know or whatever but so yes please give us feedback and I'd like to thank you for your attention at this late time of the
01:04:02
day and I hope you enjoyed it thank you