0-layered architecture
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Serientitel | ||
| Anzahl der Teile | 170 | |
| Autor | ||
| Lizenz | CC-Namensnennung - keine kommerzielle Nutzung - Weitergabe unter gleichen Bedingungen 3.0 Unported: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen und nicht-kommerziellen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen und das Werk bzw. diesen Inhalt auch in veränderter Form nur unter den Bedingungen dieser Lizenz weitergeben | |
| Identifikatoren | 10.5446/50788 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
|
NDC Oslo 2014126 / 170
2
3
5
6
8
11
12
16
21
22
23
27
31
35
37
42
43
45
47
48
49
50
52
54
56
57
58
61
65
66
67
71
74
77
80
81
83
84
85
87
88
89
90
91
94
95
96
97
98
100
102
107
108
112
114
115
116
118
120
121
122
123
126
127
128
130
133
135
137
138
139
140
141
142
143
144
145
147
148
149
150
153
155
156
157
158
159
160
161
162
163
166
169
170
00:00
EinsNeuroinformatikZeichenketteBitSystemaufrufComputerarchitekturAusnahmebehandlungTwitter <Softwareplattform>Wort <Informatik>Formation <Mathematik>Armstrong, JoeComputeranimation
00:45
DatenverarbeitungssystemSQL ServerGewicht <Ausgleichsrechnung>PunktROM <Informatik>DateisystemPunktgitterEreignishorizontKomplex <Algebra>Prozess <Informatik>Quick-SortInternet der DingeBitStreaming <Kommunikationstechnik>Keller <Informatik>PunktSchnittmengeVirtuelle MaschineMereologieBenutzerbeteiligungMAPWort <Informatik>AbstraktionsebeneMicrosoft dot netKlasse <Mathematik>p-BlockVIC 20ProgrammierungNeuroinformatikEinsLambda-KalkülCodeArray <Informatik>CASE <Informatik>RobotikDatenstrukturProjektive EbeneComputerarchitekturBefehlsprozessorUnternehmensarchitekturResultanteGewicht <Ausgleichsrechnung>HalbleiterspeicherFächer <Mathematik>SchwebungSichtenkonzeptExt-FunktorSchreiben <Datenverarbeitung>Generator <Informatik>SondierungFortsetzung <Mathematik>SkriptspracheDatenflussStandardabweichungComputeranimation
04:28
Physikalisches SystemQuick-SortTransformation <Mathematik>Klassische PhysikCoxeter-GruppeMathematische LogikUnternehmensarchitekturKategorie <Mathematik>Endliche ModelltheorieInterface <Schaltung>MathematikSoftwaretestHalbleiterspeicherGewicht <Ausgleichsrechnung>Streaming <Kommunikationstechnik>Computeranimation
05:51
ZwölfLokales MinimumVorhersagbarkeitDigitale VideotechnikSpezifisches VolumenComputerarchitekturGüte der AnpassungResultanteDifferenteSpieltheorieTurm <Mathematik>SchnittmengeMinkowski-MetrikBitKeller <Informatik>Lesezeichen <Internet>Data MiningComputeranimationBesprechung/Interview
06:41
AbstraktionsebeneGewicht <Ausgleichsrechnung>GleitkommarechnungStreaming <Kommunikationstechnik>DatenfeldProgrammbibliothekDigital Object IdentifierQuellcodeVererbungshierarchieHierarchische StrukturKlasse <Mathematik>Physikalisches SystemSpezielle unitäre GruppeAbstraktionsebeneBitLesezeichen <Internet>Twitter <Softwareplattform>Data MiningRichtungStreaming <Kommunikationstechnik>sinc-FunktionKlasse <Mathematik>Microsoft dot netMultiplikationsoperatorPhysikalisches SystemArray <Informatik>VisualisierungFehlermeldungProjektive EbenePuffer <Netzplantechnik>SoftwareentwicklerQuellcodeEinflussgrößeRechter WinkelFramework <Informatik>Schreiben <Datenverarbeitung>Computeranimation
08:33
OvalPufferspeicherLokales MinimumLesen <Datenverarbeitung>Schreiben <Datenverarbeitung>AbstraktionsebeneDatenverarbeitungssystemSQL ServerGewicht <Ausgleichsrechnung>PunktROM <Informatik>Streaming <Kommunikationstechnik>Schreiben <Datenverarbeitung>ZeitrichtungArray <Informatik>Puffer <Netzplantechnik>FehlermeldungRechter WinkelComputeranimation
09:13
Streaming <Kommunikationstechnik>Klasse <Mathematik>Lesen <Datenverarbeitung>InternetworkingServerStapeldateiClientPhysikalisches SystemSkriptspracheVisuelles SystemNormierter RaumAnwendungsspezifischer ProzessorEinsDemo <Programm>Streaming <Kommunikationstechnik>Rechter WinkelTopologieElektronische PublikationBrowserResultanteExogene VariableFunktion <Mathematik>InternetworkingZeichenketteLesen <Datenverarbeitung>GeradeGruppenoperationProgrammierungBitZahlenbereichQuick-SortSpielkonsoleSchreiben <Datenverarbeitung>Mini-DiscNichtlinearer OperatorSkriptsprachep-BlockComputeranimation
11:57
Kette <Mathematik>SchätzungBEEPDomain <Netzwerk>Konvexe HülleNormierter Raump-BlockElektronische PublikationUnternehmensarchitekturInternetworkingCodeRouterPhysikalisches SystemSatellitensystemStreaming <Kommunikationstechnik>Kategorie <Mathematik>SoftwareDatenbankNichtlinearer OperatorArithmetische FolgePuls <Technik>Computeranimation
12:49
Streaming <Kommunikationstechnik>Lesen <Datenverarbeitung>InternetworkingServerOvalZeichenketteExogene VariableE-MailInhalt <Mathematik>SoftwarewartungSynchronisierungSpielkonsoleElektronische PublikationURLServerClientSchreiben <Datenverarbeitung>AdressraumStellenringDomain <Netzwerk>Klasse <Mathematik>StandardabweichungRoutingGraphiktablettZeichenketteVerschlingungLesen <Datenverarbeitung>Wurzel <Mathematik>KonfigurationsraumTextbausteinCodeKartesische KoordinatenGewicht <Ausgleichsrechnung>DämpfungBenutzerbeteiligungWeb-ApplikationInhalt <Mathematik>Elektronische PublikationTouchscreenComputeranimation
14:24
OvalExogene VariableInhalt <Mathematik>E-MailPhysikalisches SystemFlächentheorieComputersicherheitFolge <Mathematik>ServerProgrammierumgebungStellenringStreaming <Kommunikationstechnik>ClientBitNichtlinearer OperatorLesezeichen <Internet>CoprozessorZweiBenutzerbeteiligungDatenbankPhysikalisches SystemExogene VariableVirtuelle MaschineCodeProdukt <Mathematik>Computeranimation
15:29
KreisringSpeicherabzugStreaming <Kommunikationstechnik>MenütechnikSteuerwerkMini-DiscDateisystemBrowserCodierung <Programmierung>Inhalt <Mathematik>DatentypAdressraumZeichenketteSynchronisierungOvalKanal <Bildverarbeitung>CADGeradeSpielkonsoleLesen <Datenverarbeitung>Offene MengeCodierung <Programmierung>BrowserBitElektronische PublikationMini-DiscRohdatenStreaming <Kommunikationstechnik>ClientSpeicher <Informatik>MereologieKartesische KoordinatenBenutzerbeteiligungHalbleiterspeicherDateiverwaltungGamecontrollerVirtuelle MaschineServerMicrosoft dot netZeichenketteLesezeichen <Internet>Inhalt <Mathematik>ComputerarchitekturExogene VariableNeuroinformatikKernel <Informatik>PunktTouchscreenPhysikalisches SystemBinärcodeComputeranimation
18:37
Physikalisches SystemStreaming <Kommunikationstechnik>ZeichenketteSpielkonsoleInhalt <Mathematik>BootenExogene VariableE-MailOvalSynchronisierungGamecontrollerClientSpielkonsoleBenutzerbeteiligungStreaming <Kommunikationstechnik>DateiverwaltungStellenringPhysikalisches SystemServerGraphische BenutzeroberflächeMiddlewareMini-DiscKugelkappeElektronische PublikationDifferenteProjektive EbeneObjektorientierte ProgrammierspracheObjekt <Kategorie>Quick-SortFitnessfunktionBitTouchscreenMultiplikationsoperatorCoxeter-GruppeDatenbankGeradeSpeicher <Informatik>Exogene VariableBinärcodeBrowserUnternehmensarchitekturClique <Graphentheorie>TorusComputeranimation
22:05
HochdruckAnwendungsspezifischer ProzessorVerzeichnisdienstDickePhysikalisches SystemClientVisuelles SystemNormierter RaumDatentypFolge <Mathematik>MUDZahlenbereichKartesische KoordinatenStreaming <Kommunikationstechnik>SpielkonsoleTypentheoriePhysikalisches SystemExogene VariableKugelkappeSpeicher <Informatik>MereologieBitQuick-SortTouchscreenComputeranimation
23:45
TaskSynchronisierungExogene VariablePufferspeicherOvalZählenCodierung <Programmierung>Streaming <Kommunikationstechnik>BitLesen <Datenverarbeitung>BrowserExogene VariableKomplex <Algebra>CASE <Informatik>TouchscreenArithmetisches MittelSpeicher <Informatik>ClientKartesische KoordinatenObjekt <Kategorie>NetzadresseBenutzerbeteiligungGeradeProgrammierumgebungKontrollstrukturE-MailPunktTextbausteinSchreib-Lese-KopfPhysikalisches SystemMiddlewareRechter WinkelWeb SiteFitnessfunktionFunktion <Mathematik>Gewicht <Ausgleichsrechnung>TypentheorieKontextbezogenes SystemMicrosoft dot netEigentliche AbbildungComputeranimation
26:30
Klasse <Mathematik>DatentypHochdruckMagnetooptischer SpeicherVirtuelle AdresseMUDFolge <Mathematik>Anwendungsspezifischer ProzessorDruckverlaufGammafunktionART-NetzServerStreaming <Kommunikationstechnik>PufferspeicherOvalZählenCodierung <Programmierung>Lesen <Datenverarbeitung>Dämon <Informatik>Demo <Programm>MultiplikationsoperatorBenutzerbeteiligungStreaming <Kommunikationstechnik>BitCOMGewicht <Ausgleichsrechnung>Microsoft dot netDifferenteÄhnlichkeitsgeometrieDatenstromComputeranimation
27:20
Physikalisches SystemDatenkompressionExogene VariableStreaming <Kommunikationstechnik>Visuelles SystemBrowserStreaming <Kommunikationstechnik>Physikalisches SystemDatenkompressionExogene VariableFunktion <Mathematik>ProgrammbibliothekResultanteCoxeter-GruppeBitKette <Mathematik>InjektivitätRechter WinkelKonstruktor <Informatik>DatenstrukturGewicht <Ausgleichsrechnung>SpielkonsoleGebäude <Mathematik>Elektronische PublikationComputeranimation
28:58
Physikalisches SystemClientElektronische PublikationSpielkonsoleMicrosoft dot netMachsches PrinzipGewicht <Ausgleichsrechnung>Streaming <Kommunikationstechnik>Funktion <Mathematik>MaßerweiterungResultanteStandardabweichungComputeranimation
29:40
DatenkompressionPhysikalisches SystemVisuelles SystemExogene VariableStreaming <Kommunikationstechnik>RechenwerkOvalMachsches PrinzipInklusion <Mathematik>Streaming <Kommunikationstechnik>ThreadDatenparallelitätAggregatzustandRahmenproblemDifferenteSynchronisierungMAPLesen <Datenverarbeitung>Gewicht <Ausgleichsrechnung>CASE <Informatik>BitKontextbezogenes SystemMicrosoft dot netMereologieComputeranimation
31:07
Physikalisches SystemDatenkompressionExogene VariableStreaming <Kommunikationstechnik>SQL ServerServerElektronische PublikationBinärdatenSpezialrechnerTypentheorieStreaming <Kommunikationstechnik>UnternehmensarchitekturDatenbankMini-DiscBinärcodeElektronische PublikationPerspektiveOffene MengeMultiplikationTransaktionKategorie <Mathematik>Quick-SortInformationTouchscreenVirtuelle MaschineLokales MinimumREST <Informatik>ComputerspielServerReelle ZahlBrowserBenutzerbeteiligungClientDatenstrukturDemo <Programm>Objekt <Kategorie>Exogene VariableDatensatzLesen <Datenverarbeitung>Web-SeiteMultiplikationsoperatorProjektive EbeneFortsetzung <Mathematik>Zellularer AutomatLuenberger-BeobachterHyperbelverfahrenMereologieEinsComputerarchitekturComputeranimation
34:28
SQL ServerPetri-NetzStreaming <Kommunikationstechnik>TypentheorieElektronische PublikationBinärdatenSpezialrechnerElektronischer ProgrammführerSoftwareentwicklerSystemverwaltungBrowserTransaktionCodierung <Programmierung>Inhalt <Mathematik>DatentypOvalSynchronisierungZählenLokales MinimumOffene MengeNormierter RaumComputersicherheitBewertungstheorieStrebeRechenwerkMagnettrommelspeicherInklusion <Mathematik>MenütechnikWechselseitige InformationKonvexe HülleMinkowski-MetrikPhysikalisches SystemPufferspeicherElektronische PublikationStreaming <Kommunikationstechnik>Fortsetzung <Mathematik>ServerKonfigurationsraumProjektive EbeneDatenverwaltungBitMini-DiscEinfach zusammenhängender RaumSchreib-Lese-KopfExogene VariableUnternehmensarchitekturPhysikalisches SystemDatentypKonfigurationsverwaltungBenutzerbeteiligungBrowserClientLesen <Datenverarbeitung>Eigentliche AbbildungElektronischer ProgrammführerDatenbankBefehl <Informatik>Virtuelle MaschineDifferenteTabelleSoftwareZweiProdukt <Mathematik>BinärcodeAuthentifikationDisk-ArrayBildschirmfensterMereologieKartesische KoordinatenDateiverwaltungExtreme programmingVarianzEinsLokales MinimumBildgebendes VerfahrenNebenbedingungGraphische BenutzeroberflächeRoutingMicrosoft dot netWeb-SeiteMiddlewareDichte <Stochastik>FreewareGüte der AnpassungStichprobenumfangFlächeninhaltGruppenoperationResultanteVersionsverwaltungSystemaufrufKeller <Informatik>KontrollstrukturPunktTouchscreenPi <Zahl>NeuroinformatikVerkehrsinformationÄhnlichkeitsgeometrieComputeranimation
42:06
OvalPrimzahlzwillingeDatenmodellE-MailElektronische PublikationExogene VariableZeichenketteKonvexe HülleStellenringModallogikInhalt <Mathematik>Mini-DiscElektronische PublikationMultiplikationsoperatorGraphische BenutzeroberflächeSoftwareEinfach zusammenhängender RaumBefehl <Informatik>Streaming <Kommunikationstechnik>ProgrammbibliothekBenutzerbeteiligungClientVirtuelle MaschineDifferenteStellenringPunktZweiInhalt <Mathematik>DatenbankDichte <Stochastik>ZahlenbereichFramework <Informatik>Quick-SortMicrosoft dot netStichprobenumfangDemo <Programm>VerschlingungBitExogene VariableOffene MengeGüte der AnpassungInstantiierungServerEigentliche AbbildungBildschirmfensterE-MailWeb logSpeicher <Informatik>VarianzVersionsverwaltungDateiverwaltungHalbleiterspeicherGanze FunktionSymboltabelleTouchscreenProjektive EbeneEinflussgrößeFortsetzung <Mathematik>Twitter <Softwareplattform>DruckverlaufKonfigurationsraumLesen <Datenverarbeitung>FehlertoleranzGeradeRechter WinkelMultiplikationTransaktionBildgebendes VerfahrenPhysikalisches SystemNormalvektorComputeranimation
49:41
Computeranimation
Transkript: Englisch(automatisch erzeugt)
00:03
Okay, so welcome to the last talk of the day, well except like Hanselman's talk, but he's on the, he was listed with the rock bands So it doesn't really count, the last like normal talk of the day. I'm going to talk on Zero-layered Architecture, which I called it. That's just to lure you in here, really. It's a fancy, fancy title
00:22
But I want to talk about zero and ones I heard Joe Armstrong say yesterday that there are no strings There are only ints, and I think he's right, like these are things that exist. These are the zero and ones in our computers and I want to talk a little bit about how
00:41
How we can build stuff with just a zero and ones that we have in .NET, really Just a quick few words about myself. This is me on Twitter Computas, that's where I work The hashtag CX lambda is really, I just put it in there. That's the computer club We have at work, Computas Lambda Club, so that's sort of where I play, so it's the work and the play part
01:07
This talk sort of comes out of playing and Has gotten into the work part, so it's a little bit of both. I'll come back to that I guess My current Role as the technical lead in a standard .NET project where we use, you know, stuff that everybody uses, Entity Framework, Web API,
01:27
MVC, just squeezed in a little bit of F-sharp, so I'm really happy these days But Also, I like to think around with fun stuff like robots and Internet of Things and code clubs for kids and that sort of
01:40
stuff So since this is the last talk of the day, I'm tired, you guys are probably tired and getting ready for the party I just needed to to give some structure to the talk In case I just wander off into some streams and deep into some byte arrays or something and get lost in there, so First point is that I like streams, drones, and old computers. We'll get back to that
02:04
But the bold part is like that's the key point of this talk really Without the fancy fancy title and that is I've used streams once to make a traditional You know SQL Server, Web API stack that serves up large data sets could be from wherever But it was actually from SAP which gives it gives it a little bit more of enterprise cred sort of just with massively
02:25
reduced latency and memory use compared to the normal things that we always do when we do this n-layered architecture stacks So on the way I'll try to show you all the cool streams in the .NET framework that I found and I like to think around with and just go through them and demo them and
02:43
In this talk there will be no TPL data flow, no stream insight, no complex event processing No, RX, no streams like that which are higher level. I'm just Strictly talking about everything that inherits from the abstract class system.io.stream All of this other stuff is really awesome, and it's very good
03:01
But I want to go all the way back to basic and see what can we build with just stream the stream class A few words about where I got the inspiration to do this from so the first part here on On one side here is the drone that we control that I'm playing with in JavaScript And you see this nice
03:22
piping syntax you have the node.js and large parts of my inspiration for working with streams comes from from node.js because they have a very cool story on streams, and they have a lot of talk cool talks about streams and And stuff like that and the other part is from my old machine
03:41
I just brought out my Commodore 64 that I had when I was a kid This is just like a snippet from some of the programs that I've wrote there and then we're back back to 8 bits you're working with 8 bits zero and ones and What I think is interesting is just how fast this like the computer is really really slow
04:00
But I can write programs with my Commodore 64 that are faster than some of the stuff that I write on this new i7 Fourth generation, you know 32 gigabyte RAM machine because we have so many abstraction layers And some of them block and wait for stuff to happen So the end result is sometimes slower than what they can achieve on this one megahertz 6510
04:24
CPU and that was sort of the inspiration for bringing Doing more with streams in .NET This is a cool talk. This is sort of the only talk I could find on the same topic it's Kristan Horstal talking about layers considered harmful at ODAV and
04:45
Here we see like, you know typical enterprise stack just layers and layers and layers and layers and They have probably a lot of useful properties But in this talk, he also talks about all the issues that you get
05:01
in these systems and I'm making I'm sort of restricting myself to the same systems He's talking about the classical enterprise say, you know data in data out transformations layers and decoupling and and all of that but he talks more general about it and talks about How you you know when you change something you're supposed to have like, you know decoupling
05:24
But you want to change something in your model and you know Then you need to change something in that layer and you need to change the detail We need to change the interfaces for the detail. You need to rewrite your test Oh, that was one layer next layer and like it's supposed to make it easy to change stuff But sometimes it's just a lot of work to change stuff because there's logical couplings between all these layers and and all of that
05:43
but I'm just restricting myself to talking about Memory and latency that's the stuff that I was I'm going to look at in in this presentation So I just want to show you one Video on and layered architecture a lot of big Due to sheer volume red milk pre cooks their crispy bacon every morning
06:03
stacking it high in a proud two-foot pillar of porky goodness. Oh My lord that literally is a tower Oh bacon, that's incredible How many pounds of bacon is that about 70 pounds? Set that's for today today. You do this every single day every morning starting at 630
06:23
That is like a national monument forget the spaces literally Mount bacon more Okay, you got to show me how to make this double bacon deluxe So we're really proud of this fat stacks that we're building just like yeah This guy is obviously very very fascinated by it, but I want to go a little bit more lean than this
06:42
I'm not only sitting there so it's not like I'm quoting some you know I'm just quoting a friend of mine really from Twitter The one favorite there is my favorite He said that you should fire up your abstractions and just bringing back those you need and That's again a little bit what we're doing here So instead of all these layers of bacon
07:02
I want to I want to look at just a stream of data, you know nice peaceful stream of data So another topic could have been you know, how I learned to stop worrying and love the stream so As I mentioned I will talk only about system I owe the stream class So it's been there in the dotnet framework for a long time. And there's probably lots of people in this room who's been
07:27
you know Writing stuff with streams and dotnet since before I touch dotnet. It's been there forever But that's what I'm restricting myself to so stuff that writes and reads buffers of
07:41
Bytes so zero to 255, you know an error arrays of of that That's what we're restricting ourselves to in this talk We could maybe just have a quick look at What it would look you know what it looks like like, you know, we're just just for fun, we just make a new project in Visual Studio and
08:07
You know and we Inherit the stream class is an abstract class and we implement the members
08:24
So, oh, yeah, sorry it's down here I guess. Yeah So there's some methods in here. I'm not going to go through all of them At least not now the most important stuff is here Like can I read this stream?
08:42
Like is it a readable stream or is it a stream that I write to? You know, you have the same thing for write so a stream can be readable or writable or both And then you can write stuff to it and read stuff from it and you can read and write buffers of bytes So it's you know arrays of zero to 255 wrong arrow button
09:16
So I'm just gonna start with some simple demos. The first ones are gonna going to be really really simple, but
09:22
But I want to start out just showing the tools that I'm using and so on so we're not losing anyone on the way So the first is a you know is a very is the most basic example I can't sort of believe I'm showing it here at the end to see I'm just reading all the text from a file without streams
09:41
And I'm writing it out again. So this is a typical blocking operation, you know, we read all the data in and then we write it out again And I just wanted to show it just to show how I run the program so we can run this in script CS So we can just run this program and it waits a little bit while it reads the file and then it prints everything out Okay
10:01
Nothing very interesting in there So, how would how would this look if you try to do the same thing from the internet? You know, it wasn't that big of a problem when we were reading from an SSD disk but once we try to get a string and write the result out and block on reading a
10:21
String You know, we're waiting for the response to come and that takes I don't know forever. It's a hundred and twenty hundred and thirty megabytes a big XML file, so We're not even going to wait for it. It doesn't come we have to download 130 megabytes of data before this thing comes
10:43
So, you know now it starts to get some streams in here so now we're instead of getting a string we're getting a stream and We're getting the same file and we wait for the result here So we block and getting the result it's an async operation when we do result we block and wait for it
11:03
But the result is now not all the data the result is just getting the request back and then we get the stream and That's the response stream and that response stream we can copy to To the console we don't use console write line anymore We open the standard output and that's the stream as well. So we have a readable stream
11:21
Which is the stuff we got from the internet and we have a writable stream, which is the stream You have in the console and we just copy that stuff out So if you run this We don't have to wait forever. We just need to wait for the response from the internet then, you know 130 megabytes of data starts flowing in and that's exactly what a browser does as well. Like see if I can kill this
11:47
Okay, a browser does exactly the same thing if I copy paste this stuff into a Browser I need the browser It does exactly the same thing it doesn't block. Oh that was a gzip file
12:04
It doesn't block it just starts getting it parsing it to show it to you straight away It doesn't wait until everything is downloaded. So who's the one who's the guys who are waiting for everything? That's the enterprise devs. We have like 17 layers of blocking code in our code so everything else on the internet doesn't block and But we you know our complex software we think that's so complex
12:23
We need to block all the way and we need to have DTOs and we need to transfer all the data between all the layers and so on You know for all these properties that we think our systems need but the guys are sending this stuff through I don't know what routers through 4G networks, you know through satellites and through routers and whatever on the internet Oh, they're supposed to handle this stuff a stream and then everything comes back to us enterprise devs
12:44
And then we start to block on all the operations. We start to block on database operations and we block on everything So, let's see So I want to show I think this is the last blocking thing I will show you guys I promise
13:06
But I just wanted to show you you know the the standards That I'm starting to modify off from when I move into the streaming domain So this is this is just link pad, which is a scratch pad for writing .NET code
13:21
and this starts up basically starts up a win application So it starts up a .NET web application on this localhost 8090 address and then it just waits You know for me to press a button And it starts up using the startup class and this is the configuration class down here This class doesn't really do much except telling me I'm going to use
13:44
Web API and map the attribute routes. So this is just boilerplate code for firing up the Web API So I have the Web API self-host DLL referenced in my in this file and all I'm saying is that I'm you know, I'm capturing the route on
14:02
localhost 8090 and I'm reading all the text blocking like we did before and I'm sending that out a screen content to the To the client and I'm telling the client that this is application XML so I can start this server oops, I can't I
14:21
can start this server and And Hold on and this is the client code. So this is the same stuff that you saw before but it's also blocking
14:46
so it's Doing a blocking operation getting everything from localhost and then writing it doesn't know my both my web server and the client is blocking But this is of course locally so it doesn't take forever, but you see how slow it is and then we get to response
15:02
But I think sometimes we get fooled because we're in our dev environment so we have the database on our machine We have the web server on our machine. We have SSD drives and 30 gigs of RAM and Fast processors and we're you know, we can do blocking operation that we don't really notice it because it may be half a second delay But it takes, you know five seconds to fire up
15:22
The stuff that you're starting anyway, so you don't really notice it until you get in to production Okay, so a little bit more on streams I just need to show you this one because this is my favorite stream in dotnet It's like the dev null of dotnet is system.io stream.null and this is like a null stream where you can throw stuff into
15:47
So this is the first example where you make a readable file stream and you copy it to system.io.stream.null So basically you're just throwing it away. So not much stuff happens takes 50 milliseconds and we Threw it into some dark place on our machine
16:02
But then we can do this with streams as well Why don't we get the Linux kernel and throw that away so we can go here from the kernel org and get the latest Linux kernel and We can await the request Then we tell you know this point we have the response stream then we just asynchronously throw that stuff away So we don't have to hold everything in memory. We don't have to wait for it
16:22
We don't need to fill up our valuable memory with the useless Linux kernel anything like that We just pipe everything out to stream.null We can start this and we see Oh, we have the response from the kernel and basically right now We're just throwing stuff away to reading chunks of the Linux kernel and just throwing it into some dark place in our computer
16:45
This is really my favorite Favorite stream, let's see. Okay, but we need to do something a little bit more useful. We can't serve up
17:08
We can't really serve up Files like this, let's have a look at the part of the architecture that I will like end up at it in the end and
17:20
We have web API and the file system just like we had before in this blocking application But right now we're doing I want to do streams One trick that I will show during this is that you can actually just store You know gzip files on disk serve them up as raw binary data and tell tell the client You know the content encoding is gzip and the browser has a streaming gzip
17:42
Uncompressor so that will understand that as well so we can just serve up raw gzip data We don't have to uncompress it. We don't have to compress it anything like that Let's just have a look at how that looks So we had this we had this non streaming server before Which was the web API controller that served up this XML file
18:03
So What do we need to change to make that into a streaming server So we're still using a little bit of Mickey Mouse examples here So we have a file stream so all our data is just a file on disk But right now instead of reading all the file We're just creating a file stream and web API has this nice stream content thing
18:22
So instead of returning a string content we can return a stream So now we're just piping data from file to the client. So let's start this one. I Need to stop selecting text before I click f5 so that would be
18:43
Again, we just make getting a stream from localhost and piping it out to the console So let's run this one and this is this should be much faster Takes a little while to fire up scripts. Yes, but now we have a continuous stream from the file system through web API and into the client here and
19:04
Out into the console and we could do the same thing here as well by serving up a gzipped file and Telling a client that this is gzip
19:21
So our system doesn't really know it just takes binary data and gives it to someone there is no objects There's no nothing. It's just like binary chunks of data. All we're doing is telling the browser, you know, trust me. This is gzip So if you start up, I need to stop selecting the text before I click f5 okay, so I'll start up the web server and
19:42
I'll go to local Host Lottery I haven't played a lot. Okay localhost 890 and we get the data We can have a look at this and we can see here that in the response setters, you know, it says this is gzip so Chrome understands this and it just
20:03
Uncompresses this as a stream. It's not so fun when you forget this stuff and you put in the wrong if you put in, you know, if we forget to tell Chrome this you just by accident just delete this line
20:23
then Yeah stuff just blows up But we're still a little bit of Mickey Mouse examples here because we have file streams and file stored on disk It's not like the way of the enterprise, you know
20:41
You can't store just put the files on disk and serve them up. We need to put them in databases and so on But we're coming to that a little bit later in the presentation I think I want to take a few detours into screens because I think we have time to have a look at
21:03
To have a look at some other streams that are useful We're not diverging too far from the examples But I just want to dive a little bit into to Owen middleware because we have Owen now So that sort of fits right in between Underneath web API so we can capture data before it gets to web API do stuff with it
21:22
And on the way out from web API we can do stuff with the data So we could try to do something really useful, which I think is really useful which is the Hult project that I'm working on which basically Is exactly the same server as you saw before Oops all caps server here. It's exactly the same stuff. It's a streaming server. It reads a file stream and returns a file stream
21:47
The only difference is that I've plugged in some Owen middleware here called all caps We'll take a look at what that is in a little while But that's all it is everything the web API looks exactly the same so we just plugged some some Owen magic in there
22:02
I'll show you afterwards and and We do exactly the same thing except that we've done before we get the stream And we pipe it out to the console except now. We're giving in in the request
22:22
We're giving in an application header, so we were saying we want application Jason You see I've typed it Like this, so there's a small P in here, but that shouldn't shouldn't matter so let's try to run this and This does exactly the same thing as we've all seen before you just we get a lot of data as a scream
22:48
Oh come on stop, but let's try to edit this a little bit Let's try to replace this so we have all capital letters in in our media type that we're asking for I don't think this is a
23:03
defined part of the HTTP spec The capitalization of media type headers, but in in my system it is So if you do do that, and what do we get back? Oh? All caps responses so somehow we've
23:21
Streamingly turned all the data that we're getting back into all caps how useful I don't know if you can read this fast. You're maybe not as used to working with streams as I am I can read much faster now But I stopped it for you guys to sort of catch on
23:41
So you see it's it's in all caps now. How do we how do we do stuff like that? We have the Hulk middleware that I just plugged in This is streams as well, so there's a little bit of Owen boat boilerplate around here that I'm not gonna really go into
24:01
But Owen works this way that you know it gets your requests on the way in And it gets the entire environment So we get your headers and you get your streams and your output streams everything you have access to and then you can pass this on To the web API when you're done with it So I have a few lines in here that I can explain the first thing is that I take this over an environment
24:22
Which is everything that's coming in your response stream to request streams your headers And you know your IP addresses so on and I just do this new over in context Which means I get the type you don't have to do this Then I don't need to do then I don't have to do casts and look up in object accessories I get to get a proper dotnet object. It's much easier to work with
24:44
So I take the accept header Which is the stuff I sent in with application JSON and just check I just check that it's it's equal to itself in uppercase So I'm checking you know you're asking for something in uppercase, so I'm gonna give you something in uppercase and Then we just swap out the response body in this context with a new response body
25:04
Which is the old one, but it's injected into my streams So whenever anyone further down line here the web API or MVC or whoever is writing to the response and think that they're sending Data back to the client they're actually sending stuff to my stream
25:20
And then I have the stream that they will get so I can do whatever I want to that stream And then send it back to the client So inside this stream This is just like a stuff. You know some stream. That's inheriting from again from system.io the stream and I take I take that stream that I'm getting and I save that I take care of that I make a UTF-8 decoder
25:44
So now I have a decoder and I have the stream to write to and whenever someone is writing Oh font size, sorry. Thanks for Just please remind me So whenever someone is writing to this stream, and they think they're sending the response back to the browser or whoever
26:02
First I get access to the raw bytes at the reading so I put that into my UTF-8 decoder and I basically just do a little bit of decoding in here I convert it to uppercase then I've write it back to bytes again, and I write to that underlying stream that they're getting So now we have like some streaming middleware, so you see these streams fits in everywhere
26:23
You know you can put it put them in and manipulate them at all points in this in the .NET Webstack Another little detour that I think we have room for
26:41
Is to talk a little bit about streams in OJS because as I said that's where my inspiration comes from Because in OJS they talk about streams all the time, and they do crazy streaming demos just so web rebels Last month and they had they were piping data through just
27:00
It was it was all crazy like way beyond what we're doing on the .NET framework and That's where my inspiration come from and I want to look at like What's the difference in the similarities between Node.js streams and the .NET streams? Because when I started looking at the .NET streams they felt a little bit off And I'm just going to go a little dig a little bit into that
27:24
so we can have a look at a .NET stream Thing and a Node.js thing so if you look at the .NET first that's probably the most familiar stuff to To you if I want to read the zip file and unzip it in a streaming fashion
27:42
And then write it out to the console then I get a sipped stream, and I'm not the browser so I don't know How to do this I have to do this myself, so I take this sipped stream and I need to unzip it so I need an I need some Libraries from system.io compression
28:01
So we do exactly the same thing to get the request, but we put the request inside this gzip stream And then we copy the output of that to console standard output To me this felt a bit weird because in Node.js you do a very similar thing You know this is very similar setlib instead of system.io compression
28:23
We do a get on the data when we get the response. This is the same as doing a request.result Then we're piping that response into Some stream that unzips it and we can pipe that again So this is more like you know a chain of of pipes whereas in the .NET it was a bit you do
28:44
Construction injection of streams and that felt to me a bit the other way around you're like you're building your streams the other way and I was starting to think why why are we doing this and I Have a little example. This is again a little bit on the detour of this presentation
29:02
But I found this quite interesting, so I want to show you So I tried to do the same thing as they do in Node.js in in .NET So it's exactly the same thing that we've done before we get this stream But now I made some extension methods and such so that we can pipe
29:23
Just like we do in Node.js, so we take the result we pipe it into a some gzip pipestream Which is on stream I built which is similar to this node stream So this is a readable and writable stream, and then we can pipe it to the console standard output And then I started looking at why is this you know? Why aren't they doing this in .NET because here we have a readable stream
29:47
And we get it's just injected into another readable stream that we can write out In here we need to have Readable and writable streams, so we started looking at these differences, and then we dig a little bit into this and we see
30:08
What's underneath? What is inside this gzip stream? And we start to see oh There's some shared writable state in here because now we have a stream that we're both reading and writing from and in Node.js
30:21
it's not really a problem because You know shared writable state is only a problem when you have concurrency. Node.js It's still single-threaded, so they just took away the concurrency part and it all works Whereas in .NET we have async await and things happen on different threads, so you know You need to apply locks to this and you need to design this stuff carefully
30:43
And I was able to make it run like this stuff runs it actually works, and so you can do this But there are probably dark dark edge cases in this if you use async await or if you block on somewhere or if your Synchronization context somewhere is done in a way you don't understand
31:01
so this is There is some stuff in here I think that you don't want to touch so what I've started accepting after digging into this is that the .NET framework just does it Differently you just inject your streams And you write them all backwards and you make sure that you either have like a long readable pipeline or a long writable pipeline And if not you need to sit down and think really hard
31:28
Owen we've already looked at okay, so let's talk about databases We need to stream data from databases right because I just put stuff on disk But that doesn't work in the enterprise and this stuff doesn't work in the enterprise either
31:40
I think this is the stuff that we're really looking for and using right so we need some way of get streams out of SQL Server otherwise The whole idea sort of you know we have streams now from browsers and through Owen and the web API and everywhere but we need to be able to get streams of our data stored in SQL Server and
32:04
This is the stuff I started looking for because I had a real life issue like this and that now I'm sort of getting to the what I think is the useful part of the talk the other stuff was just me talking about how cool I think streams are and I was working on a project as I mentioned before where you had to serve up huge data set from SAP
32:26
And you had this the end response in JSON was maybe a hundred and twenty hundred and thirty megabytes It was like hundred thousand employees with you know 40 properties each And there was no way to to tell the client that they couldn't get everything at once all devs are like yeah
32:40
But we just page it would just give them you know they can't get all The employees at once they have to get 100 at a time But they wouldn't accept that Excel now can swallow you know one million rows they want to do just import from web into Excel through a restful interface from web API all this data and When we have this traditional n-layered architecture, and and we're reading all the data from the database
33:04
Deserializes it into some large object structure And we hold all that stuff while the client is downloading and then we're serializing it out the request you know was like one gigabyte So that actually worked you know on a demo machine because I have 32 gigs of RAM But imagine running that on a web server, and you have the business users suddenly get access to you know all this information
33:26
And they're sitting there hitting f refresh in Excel. They would kill your web server. You know on the first the first hour so we had to look at how can we get streaming data out of SQL Server and
33:41
There was really two ways that I found It's SQL Server file screen where SQL Server server sort of pretends your data is in the database But it's not it's on a file somewhere underneath you know underneath SQL Server, but you get all the good properties of Having it in the database you can still think of it as if it is in a database you get all this transaction stuff
34:05
and multiple people can read from it and and From your perspective is in the database, but from a performance perspective is it's on a file So we can store stuff in our binary max columns And you can have open streams from it, and I'll show you some demos on that later
34:24
there are some issues with this because you need to Actually configure SQL Server to do this properly and that's very you know you have a file stream tab that you need to enable On your SQL Server configuration manager to enable this and so on and in the enterprise
34:43
That's not always that easy so on the project I was on I didn't even know where the DBAs are and I'm really really scared of DBAs And I didn't want to go to the DBA and tell them that they had to do this and this part that I've No idea where was how it's hosted. You know and that's the typical situation we all often work in
35:00
So I was and this is the stuff. I'm going to demo to you, but look and you're also you need Windows authentication authentication between Your web server and the server you Or the database server, and there are some constraints on this I think still the absolutely the best way to do it if you can, but I couldn't do it like that
35:20
I had to use Traditional SQL servers with no configuration, but you can still get streams out of them From columns that are binary like var binary max or images or var binary unity and some special data types You can still open streams directly from the database
35:40
I'm not going to talk about how to configure this and how to set it up because I'm not a DBA I've you know managed to do it on my own systems and That's good enough for me, but this book is really good It's free online and some people from Redgate to publish it You can buy it in printed versions, or you can actually download a PDF for free and everything is in there
36:00
It's very good actually The art of SQL Server file stream, I thought the SQL Server file stream is a very very like niche and In SQL Server, but it's obviously like a 250 page long book So this is the the final stack that we're ending up with I Haven't included my Hulk middleware in this I could have put that in as well, but I took it out
36:23
So this is a proper streaming dotnet stack You have the browser that can read Unpack gzip stuff in a streaming fashion you just tell the browser. It's gzip just like we did before the web API That could return streams you have SQL Server
36:42
And you have the file system underneath, but now we're using SQL Server file stream, so we're actually getting a streaming File from disk all the way so suddenly we have like a full streaming enterprise stack So how does this stuff look a little bit more cold in my?
37:14
Webapi route in here. I'll show you the database first as well
37:21
It's my fast DB, and it has like a table with two columns in it So it has just some ID and a byte array of XML So we can go and get the sipped one as you see we can't really tell the difference We don't know if this is sipped or not. We just know it's a byte array
37:42
You really need to know what your data is when you're working like this So we're just saying that I'm getting this I'm going in to get this Handle to this underlying file, that's underneath there So I go and read so all I'm doing down here is actually just reading out the file path in
38:02
SQL Server to the underlying file on disk which SQL Server takes care of for me I don't even know where it is on my computer. It's in some folder or some deep inside SQL Server and I'm returning as this response I'm going to go a little bit into how we do this because you see we don't have any using statements around here
38:21
And that's that's a story. I'm going back to Let's just put that breakpoint here and run this thing So you see now We have a file path here Which is like my machine name SQL Server some GUIDs and my database and some more GUIDs and
38:42
And this is where SQL Server puts the underlying file So what SQL Server just gave me when I was doing all this reader stuff was just like a long handle to this underlying file And then I can return a stream containing that file you just continue and we get all the data just like before and
39:05
I tried this on With SQL Server file stream. I've tried it on the proper enterprise system we have you know some rack servers with you know, 15,000 rpm spinning disks and Raids and need like a proper setup SQL production servers in noisy racks
39:23
so it's quite is it quite similar to a production system and I tried to store files of this size hundred megabytes and so on and read them through this machine which then was connected across the network and Latency I got when I did it read the data even it was raw prepared data blocking was like three seconds
39:45
Because it takes a while to read data from spinning disks which are probably normally what you're using in this SQL servers Three seconds latency even before I got the response Whereas with this it is I get the response after like 30 40 milliseconds It's just basically the network latency and I have the first byte
40:06
the only thing is as you saw I didn't put a using statement around here like you probably normally do when you work with Database connection and you dispose of them once you're finished and you're returning your data But here we're not returning all the data. We're just returning a handle to the extreme. So if I try here to
40:25
If I try here I can just try to dispose this database connection after I After I made this response And if you try that, you know It just blows up basically
40:41
Because you tell the browser like oh here is your stream. It's gzip encoded It's application XML and then you go like oh kill the connection. You're not getting anything So what you have to do is You need to wait because you can't you can't just you know, not dispose the connection either I mean you have to dispose the connection while the client is finished
41:03
But when is the client finished reading the data? It's not that hard really so I'm not going to talk a lot about it But you can just over you just need to make your own stream Which is inheriting from stream content. This is really not a stream this is the web API stream content that takes a stream and
41:22
I just override it and I take care of the sequel connection and I just tell Tell it that when you are finished with your request when this request is finished Then also dispose the database connection so we can have a look at how that works so if I run this now with a breakpoint on the dispose and
41:42
I start streaming data and we're getting a lot of data and Chrome is busy trying to render all this data while it's getting it And I kill Chrome. I Basically straight away this dispose this we're disposing the request and it just bubbles up all the way until The API is disposing its stream content and then I'm disposing my database connection. It's just something to
42:03
To think of I was scratching my head a little bit in the beginning Because you have to take away the you know The using statements which is the one normal way. I protect myself against wasting these connections
42:24
So one thing I want to try is basically showing you guys The timing of this so if you look at the network timing of getting a response here We can see the timing in Chrome We see that we're waiting for 22 milliseconds here before we're actually getting the data
42:42
Getting the data just takes forever because Chrome is rendering 130 megabytes of data in Chrome and that just that just takes a while But you see we have the first byte after 22 milliseconds Now if you compare that to some of the stuff that we made in the beginning here where we had a non streaming server
43:04
We can have a look at Look at that. So this stuff is basically Doesn't want to listen to 1890 for some reason. Let's just try another port. Okay So this is the blocking thing that we made
43:24
Oh, it doesn't even work and if you look at the timing of this request
43:44
We had to wait for one point 26 seconds And I was reading the file and serving the file from a file and an SSD disk on my local machine Imagine, you know doing this across networks. It's quite a big difference and it
44:00
Uneven When they're sitting on the same Actually, it's on the same machine if the client is on the same machine There's no networks involved at all. Still it took one point 26 seconds to read the entire file and serve it through the API and This is not even a proper layered, you know Solution that we're talking about where you're actually reading stuff maybe from entity framework and then you serializing it
44:33
As I said before it's sort of like in the dotnet framework Either we have these pipelines of readable streams or writable streams and the stuff I showed you were readable streams
44:42
So the client is reading asking for more and then web API is going to the database and telling the database You know give me some more and it's a readable stream, but sometimes you're actually producing the data on the server So this is an example from a project. I'm working on now. We're making PDFs. So we have this parsed HTML that we want to turn into
45:02
PDF but then it's the other way around this PDF library. I'm using actually is writing to a stream and The web API can do that as well. It's a little bit more I don't know fiddly to get right I don't think the syntax is a bit weird and so on but you can make this push stream content and this basically takes a
45:21
stream That pushes the content out to the client so you can do that as well But it's mostly the same thing. It's just the responsibilities have turned so Think this is more or less the end of my do-it-yourself streams
45:41
So we went from all the bacon to you know homemade DIY drinking straws Which is better than bacon So if you want more I put all the content here all the demos and even some more demos out of github This link here shows some stuff that I've made which is better documented than this stuff
46:04
Just tells you how to set up a database how to build a database Where do you get sample data if you want to recreate everything how to configure SQL Server? This is my email and my Twitter handle and my blog where I sometimes blog about these things So I don't want to keep you guys here for the party, but if there's any questions just feel free to ask yeah
46:27
We had this question I've done it in Azure like just proper Azure with Windows servers the worry Yeah, oh, sorry. I repeat if it's possible to do it in SQL Azure and We my answer is no, but I don't know I mean
46:43
I don't know, but I've tried it in what I have tried in Azure I know it's possible from blob streams and stuff you can get you can get the streams, and I know it's possible In Azure when I've done it myself like this with setting up your own VM with your own SQL Server instance obviously But the SQL Azure instance um I haven't tried
47:03
but did Maybe we could have a look at it afterwards because I'm quite interested in figuring out that as well, but I've never tried should work though Would be sad if it does it would be really sad if it doesn't So yeah
47:31
Yeah, oh Yeah, but it was the one I compared it to was where I was blocking and reading all the text Yeah, I think if you compare it to actually reading directly from the file system as a stream through web API
47:43
It would probably be be the same thing. I would be my guess so the good thing about About keeping it in the database is that if you try to do this Re-open a readable stream from a file You can only have you know one file handle to it
48:01
So we have two people trying to access the same resource And it just blows up whereas if you put it in SQL Azure it works You can have multiple reads and and all of that stuff works and You get you know transactions, and that's stuff so if you change it And you write to it again the SQL Server takes care of all of that whereas if you try to do this You're on your file system yourself you might run into some trouble
48:24
And the good thing about SQL and about the file stream is that you're not putting it in the as a blob in the database if you use that version where you're Storing it as var binary directly in the database Then you're putting large blobs in your database, and you're messing up the memory of the database and all of that stuff
48:41
but if you put it in as a file stream SQL Server just basically stores a handle to the file and Microsoft recommends that if you have more than one megabyte of blobs you should actually use the file stream Thing, but I'm guess my guess is that most of us are sending on that and that we're actually putting Larger blobs in the database and and we're probably fine with that
49:04
The only another issue though is is by putting it in SQL Server is the number of database connections I didn't really mention this, but you're holding database connections open per client So you really need to increase your number of database connections compared to a traditional solution where you can just
49:23
Switch to those faster, I think yeah anything more or are we ready to you ready to party okay? Thanks