Promiscuous: A robust service-oriented architecture framework
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
Title |
| |
Title of Series | ||
Number of Parts | 50 | |
Author | ||
License | CC Attribution - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
Identifiers | 10.5446/37486 (DOI) | |
Publisher | ||
Release Date | ||
Language | ||
Producer | ||
Production Place | Miami Beach, Florida |
Content Metadata
Subject Area | ||
Genre | ||
Abstract |
|
Ruby Conference 20139 / 50
2
3
7
13
15
17
18
19
28
29
35
39
44
48
49
50
00:00
Enterprise architectureSoftware frameworkCollaborationismReplication (computing)Level (video gaming)ConsistencyMultiplication signProjective planeTwitterComputer animation
00:32
Software developerSoftware developerWeb-DesignerArithmetic meanMassMeeting/Interview
01:00
Web 2.0Software engineeringMultiplication signBlogWeb applicationWeb-DesignerComputer animation
01:30
Software developerSoftware testingBusiness modelMultiplication signIntegrated development environmentBusiness modelCartesian coordinate systemContext awarenessPoint (geometry)Software testingMobile appCodeVideo game consoleFigurate numberLevel (video gaming)Process (computing)Data storage deviceFundamental theorem of algebraRight angleProduct (business)DataflowSound effectComputer animation
03:52
CausalityVideo gameRootTransport Layer SecurityQuicksort
04:21
Computing platformFacebookShared memoryFree productMachine visionDigital photographyComputing platformBoiling pointVideo gameTransport Layer SecurityTask (computing)Online helpFeedbackTouchscreenMatching (graph theory)Group actionPlanningReal-time operating systemComputer animation
05:31
Computer networkIntrusion detection systemSpecial unitary groupPower (physics)FeedbackSoftwareReal-time operating systemMobile appException handlingComputer animation
06:01
Group actionAnalogyThomas KuhnWeb page
06:28
Query languageEntire functionWeb pageData structureDatabaseStructural loadData structureDifferent (Kate Ryan album)Web pageTask (computing)Cartesian coordinate systemPoint (geometry)State of matterArithmetic meanMultiplication signComputer animation
07:37
Service-oriented architectureService-oriented architectureFunctional (mathematics)Transport Layer SecuritySingle-precision floating-point formatLogicMobile appMultiplication signData structureBusiness modelComputer fileComputer animation
09:01
Shared memoryTwin primeMobile appBusiness modelDatabaseMechanism designRight angleComplex (psychology)Level (video gaming)Computer animation
09:55
Replication (computing)Real numberFlow separationTerm (mathematics)Cartesian coordinate systemMobile appComputer virusLevel (video gaming)Enterprise architectureDifferent (Kate Ryan album)Point (geometry)DatabaseMessage passingComputer animation
10:40
Open sourceStudent's t-testPhysical systemOpen sourceTwitterUniverse (mathematics)Line (geometry)LogicCartesian coordinate systemData structureCodeMobile appSoftware testingReplication (computing)Point (geometry)2 (number)System callUniform resource locatorComputer animation
11:55
Business modelComputer wormCartesian coordinate systemCountingDatabaseServer (computing)Multiplication signPhysical systemMobile appDifferent (Kate Ryan album)Web 2.0NumberSystem callService-oriented architectureState observerRemote procedure callProduct (business)Computer animation
13:03
Service (economics)Traffic reportingMobile appService-oriented architectureVirtual machineLemma (mathematics)FacebookReplication (computing)Software frameworkConsistencyLevel (video gaming)State observerRemote procedure callCartesian coordinate systemLevel (video gaming)Product (business)Open sourcePhysical systemEmailReplication (computing)Enterprise architectureSoftware frameworkArithmetic meanService-oriented architectureMachine visionConsistencyBusiness modelComputer configurationMessage passingComputer animation
13:35
Enterprise architectureService (economics)MathematicsBusiness modelService-oriented architectureBus (computing)Mobile appMessage passingDatabaseDivisorComputer animation
14:07
Software frameworkSoftware testingReplication (computing)ConsistencyScale (map)Point (geometry)Line (geometry)Scaling (geometry)Goodness of fitReplication (computing)Sinc functionSoftware frameworkCodeDisk read-and-write headKey (cryptography)QuicksortSoftware developerComputer animation
14:41
Replication (computing)Message passingChainContent (media)ConsistencyEmailService (economics)Supersonic speedMilitary operationInheritance (object-oriented programming)Control flowSpring (hydrology)Game controllerMessage passingInteractive televisionInheritance (object-oriented programming)Right angleDatabaseProcess (computing)EmailService-oriented architectureHTTP cookieConsistencyKey (cryptography)FacebookQuicksort1 (number)Mobile appSpring (hydrology)Disk read-and-write headException handlingSystem callSound effectBusiness modelCausalityDependent and independent variablesMatching (graph theory)Execution unitBitControl flowBoom (sailing)Computer animation
18:15
Real numberEnterprise architectureEmailMobile appCartesian coordinate systemBitGreatest elementMessage passingSoftware developerRow (database)Covering spaceMechanism designService-oriented architectureGoodness of fitCASE <Informatik>Mixed realityMobile appDirection (geometry)SynchronizationComputer animation
19:13
Software developerBlock (periodic table)Cartesian coordinate systemField (computer science)Greatest elementSoftware frameworkSoftware testingMixed realityComputer animation
19:53
Software testingSoftware testingSoftware frameworkReal numberFactory (trading post)INTEGRALBusiness modelNetwork topologyComputer animation
20:22
DisintegrationSoftware testingSoftware testingComputer wormProduct (business)Factory (trading post)
21:00
Business modelQueue (abstract data type)Digital filterMessage passingOvalLocal ringExclusive orParameter (computer programming)Host Identity ProtocolVisual systemArc (geometry)Convex hullAdditionServer (computing)RoutingWorkstation <Musikinstrument>Menu (computing)Military operationDemo (music)ModemIRIS-TService (economics)Field (computer science)MereologyReflection (mathematics)Complex systemRevision controlAnalytic setBlock (periodic table)Reading (process)Inclusion mapOcean currentBusiness modelLetterpress printingTwitterGodMobile appComputer wormInheritance (object-oriented programming)Cartesian coordinate systemAttribute grammarBlogMessage passingSocial classInformationType theoryFacebookSoftware developerMathematicsKeyboard shortcutRight angleQueue (abstract data type)Moving averageMetadataPhysical systemCategory of beingPoint (geometry)Product (business)Computer virusOperator (mathematics)Object (grammar)Multiplication signSystem administratorDrum memoryCausalityCodeConsistencyDatabaseLaptopPerspective (visual)Network topologyState of matterClient (computing)EmailIRIS-TProper mapInternet forumNamespaceCASE <Informatik>Video game consoleTrailEntire functionService-oriented architectureTap (transformer)Game controllerComputer animation
30:07
Event horizonService-oriented architectureWeb browserCache (computing)Scale (map)ConsistencyReplication (computing)DatabaseEmailMathematicsReal-time operating systemService-oriented architectureCodeRight angleCartesian coordinate systemPower (physics)DatabaseDifferent (Kate Ryan album)Cache (computing)ConsistencyWeb browserScaling (geometry)Goodness of fitProjective planePhysical systemVirtual machineScripting languageFreewareException handlingArc (geometry)SynchronizationProduct (business)Computer animation
31:34
Right anglePhysical systemVirtual machineMobile appProjective planeProduct (business)CodeGodGoodness of fitComputer animation
32:13
SoftwareEmailBitTwitterProduct (business)Client (computing)LogicDifferent (Kate Ryan album)Traffic reportingData recoveryInheritance (object-oriented programming)Service-oriented architectureMechanism designRight angleMultiplication signHand fanCartesian coordinate systemState observerComputer animation
Transcript: English(auto-generated)
00:16
Let's get started.
00:22
Hi, I'm Karim Kadus. I'm currently CTO and founder at CrowdTap. I'm also a side project addict. You can find me on GitHub and Twitter at Karim K. So Ruby is cute. And what I mean by that is, you know, it's optimized for developer joy. I mean, anyone who attended
00:41
Matt's keynote, you know, could feel that. I personally wanted to go up and hug him after the keynote. It's so clear that, you know, the genesis of Ruby came from a place of wanting to create joy in what you do every day. Happiness in web development. You know, and Rails
01:01
arguably could be cuter. You know, Rails took those same principles of really loving what you do every day as a web developer or software engineer and taking that to the web stack. And I'm sure all of you remember the blog in 15 minutes, and that was magical because it was so painful
01:21
before to get a web application up and running. And Rails changed everything. But I'm sure as a lot of you also know, over time as your team grows and you start to actually build very interesting features, you
01:42
tend to, things start to get difficult. You know, tests start to take a long time. You know, there's great testing ethos in the Ruby community, but you know, you have a lot of tests you build a lot of features and things start to get slow. And what does that do? Instead of being able to push out code into your staging or production environment immediately
02:01
you have to wait, you know, minutes, maybe even hours before that gets in there and you lose flow. You have to context switch. That's horrible. You know, your fat models become obese. You know, suddenly, you know, you have six engineers touching your user model and people are repeating the code all over the
02:21
place because they're trying to do, you know, something that's the same but slightly different. And, you know, instead of trying to refactor the app, you know, you want to push features out, they're duplicating code. And these things become really hard to work with. And less joyful. You know, you
02:41
start to use a lot of gems and that's great. You know, that's one of the wonderful things about the Ruby community is there's a lot of us and we've contributed so much to it. But then your app starts to take a long time and, you know, you're building an interesting application so it has a lot of features and you have a lot of gems and that takes a long time and you stop experimenting. You stop just booting
03:01
up console and trying stuff. That joy of that first experience with Rails or Ruby, you know, starts to erode. And then hugs turn into yelling, right? You're in your chat room just trying to figure out what's going on and yelling at each other.
03:21
It's no longer joyful. And fundamentally what that means is you've lost your agility and I think that's what Matt was really trying to say is agility is about happiness, right? If you're happy and free to do what you want with your code, you're gonna take, you're gonna innovate. You're gonna do interesting things. You're gonna be creative. And as your app
03:41
grows though, this starts to change and you lose your agility and we're back to, you know, Waterfall but just in a, you know, same pain points but just manifested in a different way. So, you know, what was really cute, you know, suddenly is this really scary
04:02
ugly thing that has appeared in your life and you don't know what to do with it. You know, you you want to figure out a way to solve this. And so you need to figure out what the root cause is, right? That's how you solve problems. So you need to figure out the root cause and solve this problem. You want this guy out of your life.
04:22
So to illustrate that, I'm gonna go through how this manifested itself, this problem, this beast in at CrowdTap and walk through that to show how we got to where we are now. And I'm gonna start by just giving you a quick overview of what CrowdTap is.
04:41
You know, we're the leading social influence marketing platform. You know, we let big brands collaborate with their customers which inspires advocacy and drives unmatched social activity. And what all this marketing speak boils down to is we create awesome tasks, engaging fun tasks for that
05:00
brands launch that users complete to help brands market. And those tasks span the full marketing life cycle. Everything from completing a poll to taking a photo of doing something fun and sharing that on Facebook to getting free product from the brand, sharing that with friends, throwing a party around it. And we're constantly adding different tasks, engaging fun, engaging tasks so that the brands can collaborate with their consumers to market
05:24
instead of just shouting at them. And that's what our vision is for CrowdTap. So let's take a look at this screenshot and this is one of our more popular actions, which is a poll. Brands can get real time feedback on whatever they want. And here they're asking
05:42
specifically, you know, what cable networks do the users currently watch or receive. And the really powerful thing with CrowdTap is we take all this data that a user, that users generate from answering all these questions and participating with
06:00
the brand and we give the brands very rich and deep targeting ability so that not only can they target based on all the demographic data on the user, their age, income, etc., which we collect, they can also target based on everything that user has done. The entire history of what that user has done with
06:20
that brand. And they can also target based on what they haven't done. So it's a very, very rich targeting engine. But as you can imagine, this becomes a performance bottleneck very quickly because on every page load we need to know which tasks are available for a user and if targeting, we need
06:40
to, you can target based on anything that user has done, you have to load up the entire history of that user. And so, to be clear, this is a really big performance bottleneck. And this was the first pain point that we encountered. A major pain point that we encountered along with everything I was discussing. So we needed
07:01
to solve this. And the way to solve this is we need to tailor our data structure for the problem at hand. So what I mean by that is we're generating data, everything that that user is doing, and we're collecting it in a way from our main application that makes sense when you're doing just basic CRUD.
07:21
But then we want to take that data and we want to have the freedom to denormalize the data, maybe put in caching, maybe use a different database technology. We want the freedom to experiment and explore with this data to be able to solve the problem. And at the same time, we want to do it in a way that
07:40
takes us back to the joy, back to the little cute, fuzzy animals we had in our lives and not these beasts. So how do you do this? So the first thing we thought was okay, we've been reading, people have blogged about the services, what about just extracting this functionality into services? But that doesn't work. Because
08:00
even if you extract your functionality into services, what you really care about is your models. That's your data. So if you, we're not talking about extracting business logic into single purpose files, we're talking about being able to work with our data in a flexible way. And so extracting it to services
08:20
doesn't really help. And what about a Rails engine or gem extraction? Essentially this is all the same thing. It's just moving furniture around. What you really want to do is be able to mutate your data. You want to be able to play with your data structure and have freedom there. And
08:41
this brought us to, you know, what about app extraction? Why don't we extract this into a separate app? You know, we can go back to the little cute animal, because it's a smaller app, and solve all of those other pain points, but at the same time have some freedom to solve the performance problem, which was a very big business problem for us.
09:01
And so the first thing we considered is how about extracting the app and sharing the database? Well, this doesn't work, right? Because everything you need to do with your data is in your main app. Your main app opens that model. And so if you're going to do anything interesting with your data, you're going to have to do it in your main app. So what have you done? Your targeting app, the splitting up of the app is not solving anything, right?
09:23
So let's do this. Let's do a synchronous API. And this is probably what most people think of SOA, right? SOA is some RPC mechanism between apps. But again, this does not solve this problem, because if you're going to do anything interesting with your data and you want to mutate your data, you're going to have to do that in your main app. And so you're adding complexity to your
09:43
main app. We want to remove complexity from the main app and put it somewhere else, right? And deal with it somewhere else. So this doesn't solve the problem. This just adds more layers. So I hope by now it's clear what the right way to do this is. And for us it was.
10:02
We need to have a separate database for our targeting application. And we need to replicate data from our main application into this separate app and have the freedom in our main application to mutate the data as we wish, experiment with different databases,
10:21
really go back to being agile in solving the problem. And that's what we did. And it's at that point that the heavens parted and Nico fell into our laps and helped us take this architecture to the next level. So I'll just pass this on to Nico. Hello. How are you? My name is Nico. I'm a PhD student at Columbia
10:43
University in the system department. I love open source and you can find me on Twitter and GitHub at nvno. So last year I came at CrowdTap and they just extracted that targeting logic into its own separate app that we call Sniper. So that was a great win because we were able to finally see the big picture
11:03
of this application, of that logic. And so when we extracted that it was like 5,000 lines of code and since we were on that app we could really refactor it quickly because the tests were running in like 30 seconds. And we got to a point where the app was just less than a thousand lines
11:23
of code because now it got really intuitive, oh look, we should put our data structure like this. Well, what if we generalize this way and all? Which we couldn't do in the main application because everything was kind of coupled together and it was really hard to see, you know, the big picture of things. So performance went really well after that. We went from
11:43
you know, a second or two to 50 milliseconds. That was like, yay, big win. But we created another problem which was data replication. It's actually hard. So what we did first was something pretty naive. So on the main application the publisher side were like, well, let's just add some callbacks and
12:03
post the data to Sniper. And then Sniper the subscriber is going to be like, oh, I could take that data and save it to its database. But things started to break down in production. We were like, it's funny, the two databases don't have the same user count. It's weird. So we realized, wait, it's a distributed system. Things fail
12:23
all the time. So for example, if your web server right after writing its data to the database dies and is not able to do the callback, then oof, Sniper doesn't know about it. Or what if Sniper is down? Well, it won't receive some updates for five minutes. So that's a problem. So we're like, okay, so this is a
12:43
problem. And at the same time, we're looking at this, we're like oh, that's funny. That looks really like an observer, like in the Rails sense, right? Like the subscriber is really like observing the data that is changing on the other app, and we're like, well, that's a cool concept. What can it do with it? And what we really want to do
13:02
with that is to have, to split our app in a lot of different services, right? Like if we could have those remote observer and replicate data around, we're like, well, why don't we apply the same source to the email service? But we couldn't just add callbacks and see the production system kind of like blowing away like this, which is
13:22
why we introduced Promiscuous. Promiscuous is a transparent application level causally consistent data replication framework for service oriented architecture. And I'm going to explain to you what it all means. So, Promiscuous is essentially the glue between all your services. So, you can think of it as a message bus where you're going to have
13:42
services publishing their data, and then you're going to have subscriber apps and you're like, oh, I'm interested in that model and this one and that, and it's going to do something interesting. And Promiscuous sits on your models between the models and the database layer. So everything that the publisher would do, it's going to propagate those changes down to the subscriber
14:02
models without having you to do anything. So that's Promiscuous in a nutshell. And because we are really trying to get back that joy and the simplicity of Rails and Ruby, Promiscuous had to like, to provide those three key features. So first, it tackles data replication
14:22
in a novel way which scales and is actually consistent. It provides a very simple API because as a developer, like we just want to write one line of code and be done with it, and of course since we're Rubyists, we want a good tasting framework. So I'm going to go through all those three points. So data replication. It's pretty hard to
14:42
to get your head around this. But let's go through an example. So say you have some sort of Facebook app where you have users and they can send messages to each other. And, and so here like say I want to implement an email service that sends an email to whoever receives a message, right? Simple. So here, if
15:02
you would, to subscribe to the data that is interesting, you would see, oh, a user just signed up. So you create user. And then that guy just sends a message to another guy. OK, create message. Great. So when you create message, you're gonna be like, oh, fetch from my database. Oh, OK, I had that user. I can send, render my email. OK, hi, whoever. Oh, hi
15:21
and all that. But since we need to scale because, you know, we have a lot of users, we're gonna add some workers. So that's what's gonna happen. On your email service, we're gonna have, say, two workers. And now, since we need to process this message in parallel, what's gonna happen? Well, if worker one is a bit slow sometimes, maybe
15:42
it's not gonna process that create user quick enough to put in the database. And so the worker two is gonna be like, oh, who, who is that user? Oh, well, I don't find it in my database. Boom, exception. So that's the problem. And that's that's the consistency problem. But Promiscuous is going to solve this problem for you under, transparently
16:01
without you having to do anything. And so it's going to try to parallelize as much as messages that it can, but still serialize the messages that are important. For example, the create user happens before the create message. And that's what we mean by causally consistency. So as in cause and effect.
16:23
So let's see how that kind of work. So how is it doing this transparently? So when you send, so when you have a user that wants to send a message to another user, it's gonna hit your controller in some ways. And your controller, what you're gonna do, you're gonna interact with your models. So what you're gonna do, you're gonna fetch, first of all, the current
16:41
user from his cookie. And then you're gonna do a user find on his friend. And then you're gonna finally create the message with the foreign keys that, that go well. And since Promiscuous sits at your model, on your model, it's gonna be like, oh, that's what you did. So he going to infer that the message create depends on those two pieces of data.
17:01
And that's how it's able to serialize the messages that are important and serialize the ones that are not. So you could say, well, that's a bit overkill, because why don't I just retry until, you know, the user is in my database, because you would have an exception, right. But it's, it's actually more so than that. So let's
17:21
take another example to illustrate this. So say you're on Facebook, and you have people posting pictures and you want to put in an email service that send an email of the picture to all your, all of your friends. So say you have a user, consider that scenario, you have a user coming back from spring break, and he deletes his parents from his friends list, and then
17:40
he posts a compromising picture. OK. So what happens if you reverse the processing of those two messages? Well, you're gonna send the picture before having removed the parents. And you're not gonna have an exception. You're gonna have a phone call. So, so this is how, so Promiscuous really has you covered. And you don't want to, when you're developing
18:04
event-driven services, you, you don't want to, to think about all the races you could have and all that. Like, it's way too complicated. But we still want a very simple API. So, so essentially, Promiscuous is, is your main data bus, right.
18:23
And it does, so, so under the cover, you have, you have RabbitMQ, which is in a message broker. So you're gonna have publisher applications that are gonna send their messages on RabbitMQ. And then you're gonna have subscriber applications that are going to connect to RabbitMQ, and each of the services gonna have their own queue, and then going to,
18:43
to receive those messages through RabbitMQ. So no subscriber actually talked to the publisher app directly. So you're gonna have some good decoupling mechanism. And because we need to synchronize all the workers together, we use Redis. So for example, that worker is like, oh wait, so before I can create that message, I need for you to wait to create the user.
19:03
OK, I'm gonna wait for you a little bit. Oh, you died. Well, I'm, in that case I'm going to process that message. So all of that is done through Redis. So let's, let's see a bit of the API of Promiscuous. So on the top, I have a publisher application running with ActiveRecord on Park West, for example, and on the bottom I have a subscriber application
19:23
running on MongoDB. So all I need to do as a developer is to include the Promiscuous publisher mixin and publish the fields that I'm interested in. For example, email and name. On the subscriber side, I'm going to include the Promiscuous subscriber mixin and, and in my subscribe block I'm gonna put the fields I'm interested into. And that's it.
19:42
That's all you need to do. That's the entire API. That's it. So of course, once you start developing this, you want to start testing that. And Promiscuous provides you with a testing framework. So, so for us, we believe that integration tests are extremely important. And we, we have to,
20:00
to test our subscribers in a way that is decoupled from the publishers. So the publishers actually, so Promiscuous allows publishers to, to create a gem, to publish a gem with factories that represent the kind of data that, if you were to subscribe to their model, you, the kind of data that you would get. So to illustrate this, I'm going to
20:22
show you a, a, a real test of Sniper. And you can see on the top, like, we just do create crowdtap member, instead of the regular create member. And so by doing this, Promiscuous is going to take the factory of crowdtap and, and package the payload that
20:40
you would have received for real in production, and send it through your, your, your pipeline. And so you're gonna be able to test really well your application as if it was in production. So to really demonstrate how easy it is to use Promiscuous, Karim is going to show you
21:01
a live demonstration. Hello. Thanks Nico. All right, let's, let's show you
21:21
guys how awesome this is. OK, so what I'm gonna do is I'm gonna show, show you guys adding, taking one of our existing models on our, on one of our apps and subscribing to it in another. So on the, on the left side, we've got crowdtap. And I've got the,
21:42
one of our models that we're actually not publishing yet, which is package. And so we're gonna take this and we're going to publish this model and then on the, on the other side, which is Sniper, on the right side, which is the targeting app we were discussing, I'm going to subscribe to that model so that any changes to the publisher app are reflected to this,
22:03
to, on the subscriber app, and I'm also gonna put some callbacks in there to bind to any changes that are happening. So let's, let's go ahead and do that. So all you need to do first is just include, you know, the Promiscuous publisher, which Nico showed you in that API example, and then just
22:21
wrap, you know, whatever fields you want to publish in a publish block. And so in this case, you don't need to publish everything. You know, and that's important. You want to figure out what's your public, what's your API, right, and doesn't need to be the entire model. So you publish what you want. So in this case, let's publish name and features.
22:42
And that's it on the publisher side. And so let's go ahead and do the, write the subscriber. So the only thing you need to do is create a model with exactly the same name. So package. You know, we're gonna store it in Mongo. Include the Promiscuous subscriber, so, you know, reflection of the Promiscuous publisher. And then mirror
23:04
again the publish block with a subscribe block, and let's, let's only subscribe to the, the name, just for this example. And then let's print out the name every time the name changes. OK. So that's it. So that's all the code you have to write. So
23:22
let's start up the Promiscuous subscriber and take a look at RabbitMQ. So RabbitMQ has an awesome admin API UI, and we just started the subscriber. There are no queues here, and we should see a queue any second now.
23:43
There we go. So we have a queue in, in Rabbit. We're gonna kill the worker. Let's
24:03
go back to Rabbit, and this is a reason that we, we chose Rabbit is it supports persistent queues. And what that means is, even if a worker dies, your queue is still there. It's durable. And any messages that would be consumed are buffered by Rabbit, and when you start the worker up again, you
24:21
get those messages. So you guarantee that those messages are gonna be delivered. And that's a property that Rabbit, Rabbit gives you. It's awesome. So let's go back and let's, let's publish something. So first let's, just make sure we save this. Let's reload, just to get those changes. And then I want to also illustrate
24:42
how dependencies are tracked. So let's do a read. So I'm just gonna read the last member that we have in our database, and then let's actually create a package.
25:04
OK. So there we go. We've created that package on Crowdtap. Let's, let's pop over back to Rabbit and take a look at the payload of that message. And so here it is. So you can see a lot of metadata
25:21
here. Types is, you know, the class we support inheritance, so you can, you can publish an entire inheritance tree, you can publish embedded documents. If you're using Mongo, that's all supported, which, you know, is very powerful. Obviously the ID, the attributes that changed, the operation. Here's some more metadata, importantly the namespace, so
25:41
it's the name of the app. Crowdtap. Some more information helps with debugging, like the host that it was published from. Current user, if you're in a controller and you're, you're logged in, publish that. That's really useful for debugging. And then the most interesting part is the dependencies. So we did a read in, in console, and here's the associated dependency on that user.
26:02
So here's the ID and the version. We use versioning to make sure that, when this message is consumed on the subscriber side, that user exists in, in that exact state. So that's how we manage all the dependencies and do causal consistencies, and here's the right dependency, which is the, the creation, the object that we just created. So that's it. That's the
26:21
payload. So let's, let's go ahead and subscribe. And so what we should see here is, you know, hello world being printed out. Drum roll. Magic. OK. So let's, you know, let's show that, let's show update. Much easier to type
26:50
when you're in your room than in front of people. OK. All right. Sorry guys. OK.
27:28
Well, for some reason this isn't working. But we, let me try once more. That's weird.
27:45
I'm like typing. OK. Let's, if, if, if I typed any change it would be immediately propagated over to the publisher side, and trust me it works. I can show you after.
28:00
So let's, you know, so, so let's, you know, what did this do to us? So it's, we started, you know, with these two applications. The main app, which is CrowdTap, and the targeting app, which is, you know, Sniper, because it's targeting. And then we pulled out all of our analytics in Fiance, you know, Fiance
28:21
because it tracks engagements, obviously. And Bobby, you know, because he's the policeman that makes sure that, you know, when people are posting stuff that, you know, it's not spam or, and if it's awesome that it gets starred. So, you know, we can iterate on using, you know, AI techniques here to, to moderate our, our posts.
28:42
And then Paparazzi, you know, is out there scraping Facebook and Twitter and gathering, you know, data so that we can report on that for our clients. And then Casanova, you know, generates CSVs and it's Casanova because, because it works,
29:01
because it spreads the love. I still don't, I still don't get that name. But, you know, we have creative developers. And then email service, Iris, which is the, the, the notification god. And then most interestingly
29:21
is, at least in my perspective, is Captain Planet. And Captain Planet came out of a two-day hackathon that we recently had, where a developer was able to, on his, on his laptop, subscribe to the production database, production rabbit, and generate canned reports off of that data to solve a really big pain point that
29:42
we have that our account team has, and actually do that in a completely isolated way that didn't impact the production system at all. And was actually running that on his laptop, just like the fire hose of production data, on his laptop, and did that in a couple of days and worked with it. That is, that's joy, right. That's, that's back to
30:00
the joy of the fifteen minute blog. That, that's taking a really complex system and making it, you know, easy to use, and it took, you know, this and went back to something ugly to a lot of little cute, fun, cuddly apps that we, that we now have. And with that I'm gonna pass it back to Nico to wrap things up.
30:25
So to wrap things up, we've seen that Promiscos is very useful to do event-driven services. So you can trigger callbacks to real-time. For example, you want to send emails when some piece of data is changing. You want to keep your caches warm. Whenever, say, you know, whenever some piece
30:42
of data changes, you want to, instead of invalidating your caches, you can actually recompute the right thing. You could push data asynchronously to your browsers to improve their user experience. And we want to do all that also by, and also doing database replication, but different kind of databases, right. So I want to be able to replicate
31:01
my Postgres database to Lassic Search and maybe MongoDB and maybe React, whatever it is, and I want this to be decoupled, and I want this to be, do this at scale and with good consistency. So that was gonna allow you to, to really see the data that you want for the feature that you need to implement. And so real-time
31:22
is really, really important these days. And if you can embrace the asynchronous layer of your application and to really unleash the power of your data, right. So you can really innovate now. You can hire some interns for a summer
31:41
and get them to do some machine learning project directly in production without having you to be worrying about, oh my god, they're gonna screw up my system and they're gonna pollute the code and all that. Like, if you don't like their stuff, eventually you just delete their app and you're done, right. So you can really go back to being agile and try things really without having,
32:02
oh my god, I need to change so much things. Like, you get back to, to really the, to the agility of, of your team. So that's, that's about it. So we invite you to try it out. It's on GitHub, on promiscos-io slash promiscos. So we've been using it in production
32:20
for a year now and we're very happy with it, but it requires a bit of babysitting for now. So we're still working on it and we're trying to make it really a hands-free experience so that you can just plug it in and you have nothing to worry about and you have all the different recovery mechanism that kicks in at the right time and everything.
32:40
So if you feel adventurous, you can try to extract your email logic out of your main application because that's like the obvious, like, you know, you have some, this is an observer, really. And, and perhaps build some reporting service also for some, some of your fancy clients. So if you try it out, we'll be super happy to help you with the, to go through, with promiscos.
33:03
And you can find us on Twitter and GitHub, Karen K and Vino. Thank you. And now we'll take questions. Thanks.