Introduction to Watermill: Siple Go Event-Driven application in 20 minutes
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Alternative Title |
| |
| Title of Series | ||
| Number of Parts | 542 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/62005 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2023103 / 542
2
5
10
14
15
16
22
24
27
29
31
36
43
48
56
63
74
78
83
87
89
95
96
99
104
106
107
117
119
121
122
125
126
128
130
132
134
135
136
141
143
146
148
152
155
157
159
161
165
166
168
170
173
176
180
181
185
191
194
196
197
198
199
206
207
209
210
211
212
216
219
220
227
228
229
231
232
233
236
250
252
256
258
260
263
264
267
271
273
275
276
278
282
286
292
293
298
299
300
302
312
316
321
322
324
339
341
342
343
344
351
352
354
355
356
357
359
369
370
372
373
376
378
379
380
382
383
387
390
394
395
401
405
406
410
411
413
415
416
421
426
430
437
438
440
441
443
444
445
446
448
449
450
451
458
464
468
472
475
476
479
481
493
494
498
499
502
509
513
516
517
520
522
524
525
531
534
535
537
538
541
00:00
Arc (geometry)Event-driven programmingVirtual machineBuildingEndliche ModelltheorieCartesian coordinate systemWordContext awarenessPhysical systemSynchronizationServer (computing)Machine learningBlogEmailContent (media)Product (business)MereologyDiagramTerm (mathematics)Default (computer science)CASE <Informatik>Subject indexingIdentity managementBoolean algebraBitJSONUMLProgram flowchart
03:42
Execution unitPartition (number theory)Queue (abstract data type)Real numberMultiplicationDifferent (Kate Ryan album)Repository (publishing)Event horizonComputer architectureDomain nameArithmetic meanMereologyVideo gameCodeMessage passingRepresentational state transferQueue (abstract data type)Commitment schemeEvent-driven programmingCartesian coordinate systemPoint (geometry)Polygon meshNegative numberWebsite
05:53
CASE <Informatik>Standard deviationSoftware frameworkVideo gameConnected spacePhysical systemLogicLibrary (computing)Inheritance (object-oriented programming)Server (computing)
07:15
Computer programmingMultiplicationTheoryInterface (computing)Computer animation
07:49
AuthorizationComputer programmingComputer animation
08:11
Multiplication sign
08:40
Queue (abstract data type)Service-oriented architectureImplementationDifferent (Kate Ryan album)Server (computing)Message passingRight angleFunctional (mathematics)Router (computing)Interface (computing)Type theoryMereologyRule of inferenceComplex (psychology)2 (number)MiddlewareConstraint (mathematics)MetadataEmailMultilaterationArtificial lifeComputer wormQueue (abstract data type)Computer animation
11:16
Linear multistep methodCoding theoryServer (computing)Artificial lifeComputer animation
11:39
Server (computing)Slide ruleView (database)RobotComputer fileFormal grammarComputer networkGame controllerService-oriented architectureInformationProcess (computing)Perfect groupGoodness of fitCartesian coordinate systemComputer animation
12:01
Content (media)Error messageServer (computing)Context awarenessVideo game consoleActive contour modelRouter (computing)MereologyInternet service providerCartesian coordinate systemSource codeComputer animation
12:48
Internet service providerGame theoryError messageDew pointCountingMaxima and minimaStrutString (computer science)Web pageDependent and independent variablesMessage passingBoom (sailing)Menu (computing)Cartesian coordinate systemTheoryBoss CorporationImplementationFile formatError messageMathematicsGoodness of fitInterface (computing)Event horizonAlgorithmMultiplication signCountingProcess (computing)Computer wormCalculationRight angleMereologyServer (computing)CodeMessage passingLecture/ConferenceComputer animation
15:55
Router (computing)Context awarenessGame controllerService-oriented architectureString (computer science)Computer wormMessage passingError messageServer (computing)LengthContent (media)Simultaneous localization and mappingMaterialization (paranormal)WebsiteMathematicsEvent horizonMultiplication signMereologyProjective planeMultiplicationMessage passingSource codeDescriptive statisticsComputer animation
17:19
Message passingRouter (computing)Pattern languageMenu (computing)Configuration spaceData typeError messageWeb pageRandom numberStrutInternet service providerComputer wormThomas KuhnGamma functionString (computer science)LengthContent (media)MKS system of unitsCoding theoryInterrupt <Informatik>Context awarenessLocal GroupClient (computing)Instance (computer science)InformationGroup actionSelf-balancing binary search treeFluid staticsLevel (video gaming)Point cloudGoogolPay televisionReading (process)EmulatorLocal area networkInterface (computing)Event horizonLibrary (computing)Error messageComputer wormMathematicsConnectivity (graph theory)Fault-tolerant systemEvent horizonBitInterface (computing)InformationReal numberImplementationConfiguration spaceMessage passingFunctional (mathematics)Goodness of fitInternet service providerRevision controlString (computer science)Artificial lifeElectronic signatureFile formatException handlingRouter (computing)Field (computer science)Standard deviationContext awarenessRight angleComputer animation
24:59
Multiplication signImplementationBitSystem callSource codeComputer animation
25:40
Euclidean vectorMessage passingLink (knot theory)Information technology consultingComputer animation
26:06
Chi-squared distributionCodeVideo gameRepository (publishing)Computer animation
26:29
Inheritance (object-oriented programming)Link (knot theory)LaptopComputer animation
27:19
Program flowchart
Transcript: English(auto-generated)
00:07
Okay, this speaker claims that in 20 minutes Robert is gonna build a event-driven application. Well, to be kind, I gave him 25 minutes, so start your countdown clocks.
00:21
Hello, my name is Robert, and yes, I would like to show you today that we can build event-driven application in Go, and it can be as simple as building simple HTTP server. And I actually decided to put the bar a bit higher.
00:41
I think that I can do it within 15 minutes. All right, at the beginning, couple words about myself. So during the day, I work in a company named Slash ID, so I work there as a principal engineer, and we are creating some identity and onboarding solution
01:02
that is a bit more frictionless than a solution available now on the market. And during the night, I'm blogging at tridots.tech blog, where we are writing some blog posts that are covering how to create Go applications that are kind of business applications, but are also maintainable in the long term.
01:22
I know, maybe some of you had a chance to read at least one article there. There are some people. Nice, hmm, nice. I will have something special for you later. You can find me on Twitter, GitHub, Mastodon. There's also my email if you would like to write to me and ask about something. But what's the most important for today?
01:40
I'm also a tomer of watermill library, and how everything started with watermill, because I think that this is pretty important context. So a couple years ago, I worked in a company where we are creating product that, okay, we're not doing something super unusual, but the idea was that each user was able to add some content,
02:00
and he should be able to, we were storing it to MySQL, plus we wanted to have some more advanced search, plus have ability to create feed for other users with some magic machine learning models that they were doing personalization. And usually if you are building such kind of system in synchronous way, there's one problem.
02:22
So this part may be sometimes slow, because I know Elasticsearch is under high load, or I know magic machine learning model said that this day it will not work. Not nice, but it happens in work, unfortunately. And yeah, or even worse, for example, some part is not working,
02:41
and it's not best user experience if it's working slowly, or it's, so for example, you can imagine that you're adding some tweet, and you're waiting for 10 seconds, because I know Elasticsearch need to index something, or machine learning model is working slowly, or even you are not able to add this content. And it doesn't make sense, because everything will be done
03:01
on this other part of the diagram could be done asynchronously, because, okay, it's not a problem if, for example, the search, some content that was added cannot be searched, for example, for one minute after it's added, if something is done. It's much better than not allowing people to add anything. So by the book, the default solution for such problems
03:23
is using some kind of PubSub, and doing it asynchronously. So in this case, we decided to use Kafka, because, you know, it's scalable, it's nice, but, well, as usually with some concepts that you're reading in the books, or listing on the conferences, it's not that simple in practice.
03:41
And it was also the case here. So the first problem was that the big part of the team wasn't actually working in asynchronous architectures earlier. That kind of makes sense, because if you're starting to learn to code, you're not starting with building some event-driven application, you're rather creating some REST API or website. So it makes sense that it was big entry point for people
04:08
that didn't use that. And it was not the only problem, because event-driven architecture have a lot of concepts that you need to know, like customer groups, partitioning, message ordering,
04:21
at least once delivery, mesh acknowledge, negative acknowledge, poison queue. And with all of that, you need to be sure that you didn't miss an event. And it's pretty important in some domains. In some cases, okay, it's fine, you're missing some event, and okay. But, for example, I used to work in the financial domain and losing one event may, for example, mean that somebody will be not paid out.
04:42
Not nice. And in general, I believe that as engineers, we should be responsible, because sometimes the code that we are building is have really big impact to the real life. And after thinking for a while, I actually started to wonder, is it maybe something that I can do to building some kind of applications in Go simpler?
05:03
And here we are. This is how Watermill was created. So, so far, we have more than 5,000 stars in the GitHub. We have more than 50 contributors across multiple Watermill repositories. We are supporting 12 different PubSub implementations, like Kafka, like Google Cloud PubSub,
05:24
like an ATS Jetstream, RabbitMQ, but we have also some more strange implementations, like MySQL, for example, if you don't have infrastructure for some real PubSub, or, for example, would like to avoid to face commit's problem. If you are doing some more fun projects,
05:41
you can have just Go channel implementation, or WorldDB, for example. But there's one more important thing than that. Watermill has logo, and it has logo with gopher vomiting to Kubernetes logo. Don't ask me why. And you can think about Watermill,
06:00
like, so let's go back to this HTTP server example. So you can think about Watermill like something that makes your life simpler, like standard library for HTTP. So, for example, if you are implementing an HTTP server, you don't care about TLS, seven layers of network, clicking start, connection pooling, and all this stuff. You are just implementing the logic in most cases. Sometimes, of course, you may have some specific scenarios
06:23
that you care about that, but in most cases you should just implement your handlers and don't care about everything around. And as you already, some of you have shown, so I sometimes wrote the article that I think that frameworks are probably not working best in Go.
06:41
And Watermill, for example, that's the case why Watermill is actually a library. And it has pretty good two upsides. So the first one is that if you already have some system and you would like to migrate to Watermill, it's kind of simple because Watermill doesn't add anything super custom and it can be integrated with any existing system
07:01
and vice versa. If, for example, for some reason you will decide that you don't like Watermill, but you will not, so you can migrate from Watermill to some different library. So this is the good thing. And I think what's pretty important, so how everything is done, because, okay, in theory it makes some nice, but it's helping, but how Watermill is built.
07:23
And in Heart of Watermill, I would say that you can see in multiple places something that is named Unix philosophy. And it's kind of old philosophy because it's from 1978 and it's saying us to write programs that do one thing and do it well, write programs to work together,
07:40
and write programs to handle, in our case, message, because that is a universal interface. And so it's no question now. Do you know who's that? So it's Ken Thompson. So he's author of this philosophy. And what's also interesting, here's one of the authors of Go,
08:00
of Go programming language. Actually, it makes sense because if you look on the Go, for example, to IO reader or our writer, this is pretty nicely visible there. And I know that a lot of people didn't know about Unix philosophy and sometimes when I have too much time to think, I have some impression that, you know, sometimes we forgot about some good old ideas
08:21
and we are trying to reinvent the wheel, even if some problems were already solved. And you know, it may be something like in Dark Ages that it was some old, nice ideas, but it was a bit forgotten. And okay, maybe I'm thinking too much. Let's go back to the water mill. So there are a couple important times in water mill.
08:43
So the first one is message. So if we compare it to HTTP server, so it's something similar to HTTP request. So in message, we have UID that is pretty useful for debugging. We have metadata. So metadata is something like headers in a request, plus payload. So this is the place where you are storing your event,
09:01
for example. The two next important parts of water mill are publisher and subscriber. So publisher, you can publish those messages and with subscriber, you're right. You can subtract for those messages from the provided topic and receive that by the child. Usually, we are not using these interfaces because it's used somewhere internally in water mill.
09:23
But for example, if we would like to add a new implementation of PubSub, this is something that they're implementing. And each PubSub implementation is implementing this interface. That's why I actually like this interface, because it's making some constraint on the implementer that okay, they need to implement that in that way. But it's also not good
09:41
because it's making each of them pretty compatible with themselves. And then last but not least type is handler function. The handler function is something like HTTP handler that you are implementing in your HTTP server with the small difference that instead of receiving HTTP request, you are receiving message
10:01
and optionally you can receive the message. So the idea is that you can react on some message, do something and emit some other messages so you can do some kind of chaining later. I will show shortly an example. And everything is magically connected. Sorry, it may be even small, but you need to trust me that in the middle
10:20
there is a router here. And this is connecting everything. So message is going from some publisher. It doesn't need to be water mill. It's going to the queue by subscriber, the router. Router is passing it through middlewares. Middlewares work in water mill like HTTP. So another thing that is pretty similar.
10:41
And it's processed by handlers. And later if we want, we can publish some other messages. Not super complex. So do you know the first rule of live coding? Don't do live coding. So do live coding. What can go wrong?
11:00
All right, so I need to change sharing settings. So one second. I don't, probably not this one.
11:28
This is why you are not doing live coding.
11:40
Yes, okay. So something does work, that's good. But not really like, I want it, perfect. This is something that I wanted to have. So I prepared a simple application here.
12:03
And what this application does. So if you are not from Brussels, so this may be something familiar to you. So it's allow you to book a room in hotel. So you can provide room ID, plus guest counts, and let's see if it works. Okay, it seems that it's not working sometimes.
12:21
Sometimes it's working, sometimes it's not working. Sometimes it's working slowly, slowly, slowly, slowly. Sometimes it's even not working slowly. So it's even worse. So let's check the source code of that application. So okay, so here we are running HTTP.
12:41
So boring, signals handling, boring. But this is probably not boring. This is usually when the most interesting part of the application lives. Let's check our handler. So okay, so we are unmarshaling stuff, to book room request. We have some advanced algorithm of calculation of room price.
13:01
And we are taking payment. What can go wrong here? And okay, as we can see, our payment provider, it's not super stable. But okay, I don't know, let's imagine that it's our boss colleague and we cannot change that politics. It happens. It's okay, what we can do? So in theory we can do like that.
13:21
Go, fine. Okay, done, it works now. But it's one problem with that. So if our server will die, there is a chance that we'll not take payment. And it doesn't like that as the best idea. What will be my idea? Instead of doing it synchronously with this HTTP handler, I would like to emit some event,
13:41
listen to that event, and take payment asynchronously. So let's do that. And let's do that with watermill, of course. So at the beginning we need to get rid of that. And we need to have our publisher here. Message publisher. So this is the interface that you should remember.
14:03
All right. And I also can prepare some code snippets to not lose time on some boring stuff like room booked. Boom, we have our event. So. Room booked, right.
14:25
Yes, count. And price, room price. All right, now we need to marshal that because we are sending bytes between our processes through our pops up through JSON. Because JSON is kind of common and it's pretty easy to debug. So let's marshal that.
14:44
Payload error. Room booked. Don't do such error handling at home, please. And now let's publish count.
15:01
So H, publisher. Publish topic. So let's use bookings. And we need our message. So let's remember we need to have UID. So it doesn't matter actually what format of UID it can be. I know, it can be even empty for some implementations
15:20
but good luck with debugging. And room booked payloads. All right, and it returns error. So we need to handle that in a nice way. But it's live coding so it's fine. All right, so we have the first part. So we have our room booked event. We're publishing that to the topic bookings.
15:42
And okay, so we just need to inject now the publisher. So let's check where it's created. Okay, we no longer need payments. I heard that Kafka is nice and scalable. So let's use Kafka. I have also snippet for that. It's nothing magical here. It's just what this and the water mill documentation.
16:01
And let's use this publisher. We don't need subscriber yet but probably we'll need it later. All right, by the way, I'm running some nice Docker Compose under the hood that is recompiling the project each time when I'm putting changes there. At the end of the presentation, I will give you materials with all the source code and with the description how it's done
16:22
that it's automatically reloading after each change. All right, so we have our publisher. We are publishing our event. So let's check if it works. Hopefully it will work. Okay, so you can see that our API is pretty stable. And let's check if our event is really published.
16:41
So we'll use mill tool. So mill is part of water mill as you can guess and we'll consume from bookings from Kafka. Mill is allowing you to consume messages from multiple websites that are supported in water mill. I know that there is tool for that in Kafka but you know, it's not mine. And yeah, with mill you can use multiple websites.
17:03
And okay, as you can see, now we have event here. So it seems to work. Okay, so done, thank you. Not really. We are not taking payment. So probably if our company will go bankrupt pretty quickly. So we need to start to take payments. So for that, we already have our subscriber.
17:22
That's good, so let's uncomment that. Okay, we need to have water mill router. So I'm gonna search router error. Router config, water mill logger,
17:44
good error handling. And now we need to add a handler. So we'll use add handler. So we'll need to provide handler name. So it will be payments. It doesn't matter really what is the handler name. Again, pretty useful for debugging. Substrate topic. So we're subscribing to the topic
18:01
that we published this message. So this is bookings. Bookings. We need to use subscriber and we need to publish the topic. So we'll publish event when we were, when we succeed to take payments. So payments,
18:22
publisher, and handler function. So hopefully you remember handler function signature. So yeah, we are receiving message and we are returning message. But we'll do it in a bit more fancy way. Payments handler because we can inject some dependencies earlier.
18:42
I need to fix that and that. Right. So we have our payments handler. So we'll receive message and we'll take payment and emit some event. So we need to have our payment provider. And what?
19:01
We need to have room booked. We need to un-marshal that. So message payload to room booked error. And compared to standard library HTTP handler,
19:24
you can return errors from a watermule handler. So I don't need to panic. And all right. So we should have the payload that we published here. So that's good. So we can now use that to take payment for room booked price.
19:43
Great. Great. And as I said, so I would like to also emit some event. So it may be useful. So if we want to emit event that we took the payment, we can have some BI or we can, I don't know, do something else. I mean, either we can send beer to this person after he booked room. Because why not?
20:04
Okay. So we need the second event. Payment taken. Payment taken. The fields. Room booked.
20:22
Room booked as well as price. And we need to marshal it again to JSON. Cool. Okay. And the last thing that we need to do is returning message.
20:43
Message as new. Message new. UID new string. And payment taken payload. I hope that I'm not writing too fast or too slow. All right.
21:00
So in the order, there is a chance that it may work. So what we're doing. So we're receiving our room booked events. We're marshaling that. We're taking payment. And when we succeed, we are emitting another event. Sounds like a done. So the only thing that we need to do is to reuse that tender. So we have that one.
21:23
And tender. Cool. Let's check if it compiles. It even compiles. So let's check if it's working. So let's look couple rooms. And the idea is that by default, water mill handler will try if the payment provider failed.
21:43
So in there, we should see some information that payment was taken. And we don't see that, huh? I know why we don't see that because we didn't start at router.
22:02
Context error. It's a bit naive implementation because it's not really graceful shutdown. But in the documentation, as I remember, we have examples with real graceful shutdown. Okay, and let's see. Okay, so we have some random error.
22:20
And you can see payment taken. Hooray, our company is saved. All right, so this is working. But there's one problem with that. So now we figure out that, okay, actually Kafka is a bit hard to run. And we are on GCP, so maybe we can use Google Cloud PubSub. So I think that I can change Kafka implementation
22:43
to Google Cloud PubSub in one minute. I'm rewriting the bar today, hi. But I think that I can do that. Let's see. Let's start the timer. One, two, three, go.
23:15
Okay, let's check. I think I did that. So let's book.
23:20
And okay, payment taken. We can double check. So let's use Neo. And let's consume some bookings. You'll see, it works. All right, so it will be that from live coding.
23:41
One last thing that I would like to show you, because you might notice that, okay, it's a lot of boring JSON there, et cetera, et cetera. You might notice that I don't like boring stuff because probably there are more interesting things to do than marshaling to JSON. So that's because, because of that, we created a component that is named CQRS component.
24:03
And the idea is that instead of doing this JSON Marshall and all that stuff, you can provide configuration to which format you would like to marshal everything, and under the hood, it would be done. So you can use, of course, JSON. You can use Protobuf, Avro, I don't know, even something custom, if you really want.
24:21
The idea is that you're only implementing this interface, so you're providing the name of the handler. You are providing the event that you are expecting to receive, so in that case, it will be room booked. And you may notice that it was pre-generic, so we have plenty interface here, but we are working on the newer version.
24:41
And you are just receiving this event, zero, un-marshaling, or whatever. And the same is going when you are publishing an event. So you are just providing the struct, and Watermill under the hood is doing all the marshaling stuff. Okay, so I think that will be all for live coding.
25:01
It looks that I was lucky this time that everything worked. And yeah, of course, it's still not production-grade implementation. I mean, it's even hard to create a production-grade implementation of HTTP server, so it's more kind of inspiration to look deeper and see that, okay, it's not that scary, but you need to take into consideration
25:21
that there are things like Kafka and Google Cloud, PubSub internals, like this one's delivery, actually shown the securest component, but I didn't call that, but it's helping a bit. So where you should start, because okay, it may be a lot of sources for you and a lot of stuff to check. So I heard that we have pretty nice documentation,
25:42
so we don't have any consulting or whatever for Watermill, so we kind of don't care to have bad documentation. So yeah, I heard that we have pretty good documentation, so at the end of the presentation, it will be in the link. What else? We have also a lot of examples in Watermill,
26:02
so I will encourage you to, oh, it's black. Oh, life code, okay, it's not life coding. So see, not only life coding, it's risky. So yeah, we have a lot of examples that probably you cannot see because it's on the block, but you need to believe me that this is on the Watermill repository.
26:21
At this point, I wanted to say a big thank you to all Watermill contributors, because without you, it wouldn't be like it's now. And ah, and small announcement that we actually released Watermill 1.2 after having too many release candidates. So yeah, finally it's released, and you are all invited to online release party, when we will say what are the new features,
26:43
and it will be on March 1st. On the last link, it will be also linked for that. And I think that will be all. So this is the, again, it's not working. Oh, yeah, so this is the link that I promised to give you. The bonus that I have, I have super fancy holographic sticker notes. I'm sure that you don't have sticker notes,
27:01
laptop stickers. So I'm sure that you don't have holographic ones. So if you don't have, so I have a lot of them. And yeah, I think that will be all. So thank you very much for your attention. Thank you.