gRPC, Protobufs and Go... OH MY!
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 561 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/44305 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 201924 / 561
1
9
10
15
18
19
23
24
27
29
31
33
34
35
38
39
40
43
47
49
52
53
54
55
58
59
60
63
65
67
69
70
78
80
82
87
93
95
97
102
103
104
107
110
111
114
116
118
120
122
123
126
127
131
133
136
137
139
141
142
148
153
155
157
159
163
164
168
169
170
171
172
173
174
181
183
185
187
188
193
196
197
198
199
200
201
205
207
208
209
211
213
214
218
221
223
224
226
230
232
234
235
236
244
248
250
251
252
253
255
256
257
262
263
264
268
269
271
274
275
276
278
280
281
283
284
288
289
290
293
294
296
297
300
301
304
309
311
312
313
314
315
317
318
321
322
327
332
333
334
335
336
337
338
339
340
343
345
346
352
353
355
356
357
359
360
362
369
370
373
374
375
376
377
378
383
384
387
388
389
390
391
393
394
395
396
406
408
409
412
413
414
415
419
420
425
426
431
432
433
434
435
436
438
439
440
441
445
446
447
448
453
455
457
459
466
467
471
473
474
475
476
479
480
484
485
486
489
491
492
496
499
500
502
505
507
508
512
515
517
518
529
531
533
534
535
536
539
540
546
550
551
552
553
554
555
557
558
559
560
561
00:00
System programmingBuildingComputer configurationPrototypeTwitterInheritance (object-oriented programming)QuicksortDifferent (Kate Ryan album)Computer configurationGroup actionBlogVirtual machineClient (computing)Goodness of fitMultiplication signSpacetimeServer (computing)Computer animationLecture/Conference
02:16
Buffer solutionCommunications protocolStandard deviationStreaming mediaControl flowDataflowComputer programmingProxy serverBlogImplementationServer (computing)Client (computing)Interface (computing)Loop (music)SCSIInformationLoginData storage deviceSoftware testingStorage area networkBuildingService (economics)VolumeLogicElectronic mailing listString (computer science)Inclusion mapContext awarenessStreaming mediaCuboidServer (computing)Volume (thermodynamics)Parameter (computer programming)iSCSI2 (number)LogicService (economics)Inheritance (object-oriented programming)Storage area networkCommunications protocolReverse engineeringData storage deviceQuicksortRight angleComputer fileMiniDiscElectronic mailing listSpacetimeKeyboard shortcutInterface (computing)LaptopClient (computing)CodeProxy serverDevice driverDataflowProjective planeBuildingGame controllerProfil (magazine)ImplementationGoodness of fitCursor (computers)Computer configurationInformationComputer animation
06:37
Gastropod shellProgrammable read-only memoryKeyboard shortcutMultiplication signDirectory serviceCodeRight angleReal numberComputer clusterData structureVolume (thermodynamics)NumberCuboidCommunications protocolMessage passingParameter (computer programming)Client (computing)String (computer science)CompilerService (economics)Computer fileServer (computing)PasswordProfil (magazine)Virtual machineCASE <Informatik>Group actionDependent and independent variablesComputer animation
12:23
Normed vector spaceGastropod shellMessage passingVolume (thermodynamics)Context awarenessDemo (music)Software testingComputer fileCodeIP addressServer (computing)CuboidClient (computing)Service (economics)Partition (number theory)Physical systemLine (geometry)System callTerm (mathematics)LaptopKeyboard shortcutInheritance (object-oriented programming)Dependent and independent variablesMusical ensembleRight angleReal numberMultiplication signCompilerFunctional (mathematics)Computer animation
18:10
Computer animation
Transcript: English(auto-generated)
00:05
My talk is gRPC, Protobufs, and Go, oh my. So how many of you have already used gRPC and Protobufs? OK, so a bunch of you. This might be a review for you. This is kind of like an intro, a sort of getting started, and so we're going to go through.
00:21
I'm John Griffith. There's my Twitter, there's my GitHub. Reach out to me after this if you guys want to chat, if anybody has any questions, or wants to tell me I did something wrong, or whatever, or just wants to meet me and talk. I work at Red Hat, and I'm mostly around the Kubernetes space these days. So, RPC, basically we're just talking about machines,
00:41
talking to machines. The whole gRPC thing and everything, it's not necessarily a groundbreaking new idea. This technique and these kind of ideas have been around for a long time. I'm old enough to have worked with most of these, unfortunately. So, gRPC is kind of the newest take on this idea.
01:04
Simple idea is basically you have some clients that you want to be able to actually affect actions on a remote server. Usually that transport that you're going to use is TCP. One of the nice things about gRPC is that it's going to use HTTP2 specifically, which gives us a lot of flexibility and a lot of capabilities
01:23
and much better performance and efficiency. So, my talk really is actually just to tell you that gRPC is really the only option out of all the different things that are out there, and it's the only thing that you should use. So, that's really why I'm here. Okay, not really.
01:41
Actually, I'm the opposite. There's a lot of really good options out there. There's a lot of really good technologies, a lot of different things you can do. But gRPC is actually really cool. It's got a ton of super, super good features, things like that. So, what I always tell people is just please always investigate new things
02:03
and look at other options out there when you're doing your research. Don't just listen to what somebody said at FOSM or anywhere else, right, or what they wrote on their blog. Check it out for yourself and see what you think. So, some of the things that I think are really nice about gRPC is
02:21
you get pretty much a complete solution out of the box with gRPC. So, I lump protobufs into gRPC because they're hidden when you're actually trying to do an implementation or build this. So, these protobufs, it's got some really cool advanced features that we might get into today, hopefully, like streaming.
02:44
So, you can even do bi-directional streaming. So, you can have a client and server stream back and forth, open a channel. Blocking and unblocking options, flow control, all the sorts of things you would expect. And one of the really cool things is the ability to build in with protoc, a reverse proxy for rest.
03:00
So, if you're hung up on rest and you need a rest compliant API, you can actually still do that. You can actually use things like protoc to build that. So, that's really cool in my opinion. But the main thing, the reason I love gRPC and I started using it a while back is because it's super easy and I'm lazy. So, all right.
03:22
So, basically building a service with gRPC, it distills down to just three steps, right? So, the first thing we're going to do is we're going to create a interface definition. So, that's our protofile and that's going to just be saying what it is that we're going to serialize back and forth across the wire. The second step is to actually implement the server
03:41
and then the third step is to implement the client. Now, I did leave one important thing out of here, right? You have to have an API of something that you're building, right? To do something useful. So, but that's something you've probably already done. So, my argument is that it's so easy that we could actually
04:02
just build a gRPC service right here and now. And not just the usual ping pong, you know, anything like that. So, what we're going to do today is we're going to take this simple LVM package that I have, the simple LVM API and we're going to build a gRPC service for it right here on the fly in this session.
04:22
So, it's going to be really interesting because this is not my laptop and it has a UK keyboard. So, you guys should be entertained, if nothing else. All right, so what we're doing is, you know, you should always kind of have an idea of what you're trying to do.
04:41
We're going to basically build our own SAN device. So, I'm doing a lot of stuff with Kubernetes and containers and all these different things in the storage space and I don't want to go out and like buy a storage box from somebody or something like that because I left those startups and I don't do that anymore. So, what I want to do is I want to take LVM
05:01
and I want to put LVM on a server that I have with a bunch of disks and then use iSCSI and just do exports and just build my own SAN basically. I actually started to do this as a project for Kubernetes for iSCSI D driver that's in there and found that all of the code that's there is actually deprecated and obsolete so I'm rewriting it and this is what I'm doing.
05:23
Okay, so, I've got this package, it's called go-lvm and we're going to start off really simple. It just creates logical volumes, lists logical volumes and deletes them. So, everybody knows what LVM is, I'm assuming. All right, so, step one, I'm going to walk through the steps again real quick
05:41
and just give you a preview of what we're going to do. So, we're going to define our interface, right? So, that's going to be our proto file and then we're going to implement a server real quick and then we're going to implement an actual client and then we're going to deploy this and run it and check it out.
06:01
So, we'll start off and we'll just make this really simple. We'll just do like create volumes. So, I'll try not to talk too much while I'm... Well, that's not going to work. Who am I kidding? Yeah, okay. I can talk really loud. So, if I can find my mouse or my cursor.
06:21
All right, so, this is going to be good. Let's minimize this and get this over here. All right, so, I've got a couple of Fedora boxes running on VMs
06:45
that I'm running on this machine. So, what I'm going to do is I'm going to go on to those and let me actually mirror this because... Well, we'll just try it. Okay, so the first thing I need to do is I am in my go path. I'm in my GitHub directory.
07:02
So, if you look PWD, you can see there's Jay Griffith. Can you guys see that or should I make it bigger? All right, so we'll do that. So, I'm going to set up my directory structure real quick and the way I'm going to do that is I'm just going to say I'm going to create a directory for my proto files,
07:22
a directory for my server and a directory for my client. Okay, we're quiet. Oh, see, and there's the UK keyboard right there. Oh, no. All right, relax. Don't panic.
07:40
Here we go. We can recover. Okay, so, let's go into the proto file. The proto file is actually, this is really kind of the nuts and bolts, right? This is the main thing. And what we're going to do inside of here is we're going to create this file and we're going to call it lvm.proto. And what this is going to do is this is going to define
08:01
what we want our service to be, right? So, let me see here. So, we start off and we just say... Okay, so syntax.
08:22
We're going to use proto3 as our syntax. So, this just tells the proto compiler what protocol we're using in this definition, right? So, this is what we're looking for. The next thing we need to do is we need to give this a name, right? So, we use kind of a goish syntax here
08:41
and we say package and we're going to call it lvm. And if I'm typing things or spelling them wrong, don't holler at me because I can't see. So, the first thing we want to define is this thing called a message, right? So, a message is the actual thing that we're going to serialize and send over the wire, right? So, anything that you want to serialize and send over the wire,
09:01
it's basically called a message. So, we're going to create a message. We're going to call it a volume. And hit capsule lock instead of... So, we're going to call it volume and we're going to give it a string called name. The next thing you'll see is this one.
09:21
One is the parameter. So, what you do is you order the parameters that you have. So, it identifies that parameter inside of the message and that's what we have there. The next thing we want is our actual create volume. That's not even close.
09:46
Hey, I could get used to this keyboard. Okay. Create volume request. So, string, we're going to give it... To do a create, we're going to need a name.
10:03
And we're going to need... If you know anything about LVM, you're going to want a volume group. And in our case, we're going to use thin provisioning. So, we're going to want a pool. We're going to want to specify a pool. And then we're going to specify a size.
10:31
Num, num, num, num. All right. Okay. So, we've got that piece.
10:43
The next thing we need to do is we need to have a request, right? So, we do a message, create volume request. I'm sorry. Yeah.
11:06
Holy cow. Off the rails. All right. So, we're going to give this another minute and then if things don't get better and people keep leaving... Not cool, by the way.
11:22
Then what I'll do is I'll just go ahead and bring up my code. And we can just kind of walk through it instead. Okay. So, there we have a response. We're just going to call it. It's just a message, right? So, the last thing we need to do
11:44
is to find the actual service, right? So, we'll do like that.
12:09
And then we want to do the same thing. I almost got it. That's not terrible.
12:22
All right. So, here's what we're going to do because we're going to run out of time. What I'm going to do instead is I'm going to pull the code because I already wrote it. And we're going to look at it. It's so easy to do if you have a non-UK keyboard when you're a stupid American
12:42
who doesn't have his laptop, their laptop. Okay. So, I'll just use magic.
13:19
Voila, right?
13:21
Okay. So, if you look back, remember our proto file that we were building. Oh, stop that.
13:40
Oh. And he fell apart.
14:00
So, that's the proto file that we were trying to build. Those are the last two lines I was trying to enter there. So, the next step that we needed was we needed a server, right? So, if you look here, I have a pretty simple server that I put together. Just standard go. We have a main.go here. What we did is we went ahead
14:21
and we defined our function for create volume. So, you can see there we take a context. All of our GRPC stuff is going to pass context back and forth. And then there's that create volume request that we had. So, what we did with that proto file is we actually ran the proto compiler, the proto C compiler, and we compiled a pb.go file. And that's what I'm referencing there,
14:41
that pb.create volume request, right? So, that's where that's coming from. So, you can see we have the create volume request, we have the create volume response. And then I just go ahead and the server is actually where LVM is installed and everything. So, I'm just executing the LVM commands on that server when it gets this command. So, that's how it's working.
15:05
One of the things that I say that's super nice about GRPC and one of the reasons why I like it so much is in terms of setting up the server and get everything wired together, this is basically it. Unless you wanted to do some more advanced things, which you can, but this will get you a basic running server
15:21
and everything will work, right? So, it's pretty simple. All right, so we've got that. And then we would need a client. And the client is even easier. We just put basically the calls into our client.
15:40
We're going to just do this directly right now. We're going to go ahead and wire in our IP address for the server. And we're going to go ahead and just use this hard-coded create volume request, okay? So, if we look here, so if we do a pseudo-LVS, so right now you can see
16:02
we've got the system partitions and we've got that thin pool for LVM thin provisioning. And that's it. So, what I'm going to do right now is I'm going to kick off this server. Okay, so now the server, the gRPC service is up and running
16:21
and listening on this server, right? So, we can go over here. I have this other box called FETI2. And what we're going to do over there is we're going to run the client. And I'm already there, I think.
16:41
Okay, so this is the same client. It's all the same code and everything else. I just copied the whole thing over here. So, what we do is we do a client. We run that. Uh-oh.
17:00
Did I not pseudo that when I started it? So, very important. Okay, so we get our message says it was created successfully. So, we'll go over here and we'll do,
17:27
and you can see we have a new LV called test. And that's the demo that I was trying to code on the fly. I have a much greater appreciation for Just for Funk now.
17:43
I'm sorry, what? Oh, yes. How's that? Is that better? So, I'm about out of time, but real quick, anybody have any questions?
18:01
Alright, thank you.