Hermes Message Dispatching
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 97 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/45709 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 201066 / 97
1
2
4
5
6
8
14
15
16
17
23
29
38
41
42
44
46
47
48
50
53
54
62
63
64
65
66
71
74
75
77
78
79
80
82
84
85
94
00:00
Physical systemSocial classMessage passingLecture/Conference
00:28
CASE <Informatik>MereologyProjective planeDistribution (mathematics)BuildingAreaPhysical systemComputing platformService (economics)Hermite polynomialsDatabaseMultiplication signLecture/Conference
01:52
EmailInformationVirtual machineMultiplication signSelf-organizationLecture/Conference
02:28
EmailSoftware bugVirtual machineBuildingArithmetic meanPhysical systemComputing platformDistribution (mathematics)Lecture/Conference
03:11
Self-organizationEmailLecture/Conference
03:33
Physical systemMessage passingConfiguration spaceEmailEvent horizonData storage deviceDecision theoryInformationSystem administratorMereologyComputer fileImplementationClient (computing)Moment (mathematics)Virtual machineBuildingSoftwareGame controllerDistribution (mathematics)CASE <Informatik>Phase transitionLecture/Conference
06:03
MathematicsWeb browserSource codeInformationTwitterBasis <Mathematik>InterprozesskommunikationBuildingOpen setBitSet (mathematics)EmailProjective planeService (economics)MereologyArithmetic meanMessage passingLine (geometry)Physical systemFactory (trading post)Ring (mathematics)Lecture/Conference
09:11
Service-oriented architectureData storage devicePhysical systemWeb applicationClient (computing)Message passingScripting languageFitness functionCartesian coordinate systemEvent horizonMereologyMoment (mathematics)Radical (chemistry)Configuration spaceWeb 2.0Lecture/Conference
10:37
Cartesian coordinate systemDatabaseMultiplication signMoment (mathematics)Lecture/Conference
11:17
Message passingBuildingInterface (computing)String (computer science)System administratorKey (cryptography)Radical (chemistry)Type theoryClient (computing)Parameter (computer programming)Physical systemService (economics)InformationMoment (mathematics)Scripting languageComputer-generated imageryEvent horizonStreaming mediaComputer architectureComputing platformWater vaporLecture/Conference
13:38
Connected spacePasswordType theoryPay televisionMessage passingMathematicsLecture/Conference
14:30
Virtual machineStructural loadMessage passingCausalityServer (computing)EmailLoop (music)
15:08
Filter <Stochastik>Pay televisionCartesian coordinate systemDiagramInterface (computing)Expert systemMathematicsAbstractionAsynchronous Transfer ModeRuby on RailsUser interfaceOffice suiteMessage passingLecture/Conference
16:02
Pay televisionMessage passingView (database)Reading (process)Physical systemDatabaseClient (computing)MereologyEmailMultiplication signWeb applicationTable (information)Parameter (computer programming)DiagramSet (mathematics)Filter <Stochastik>Formal verificationQueue (abstract data type)
17:13
Projective planeFreewareWeb pageServer (computing)BuildingOcean currentOpen setProduct (business)Independence (probability theory)SoftwareSource codeDatabaseCodeHypermediaService (economics)Scripting languageMereologyWikiLecture/Conference
18:45
Physical systemMessage passingConnected spaceScripting languageMultiplication signClient (computing)SoftwareSemiconductor memoryImplementationString (computer science)Fraction (mathematics)Parameter (computer programming)Functional (mathematics)Perfect groupInternetworkingNatural languageRight angleMetreConfiguration spacePower (physics)Hash functionOpen sourceEmailRadical (chemistry)Module (mathematics)Lecture/Conference
Transcript: English(auto-generated)
00:00
Our first speaker today is Klaas Feitag. Sorry, I don't remember your name. He's going to talk about Hermes message dispatching, as you can see here, which, as I understood it, is a system meant to have several struggles to talk to each other. So it will be very interesting, and which you guys will see soon. So please, everybody, welcome Klaas Feitag.
00:23
Thank you. Thanks. Thanks a lot. So yeah, my name is Klaas Feitag, and I work for SUSE now more than 10 years. And over the time, I was doing a lot of projects in the tools area that we use at SUSE,
00:43
how to build and databases we used for stuff to build packages and distributions. And now I'm part of the so-called OpenSUSE boosters team. We try to do activity to enable the community better
01:02
to work with OpenSUSE, use OpenSUSE as a platform for the upstream and project activities. Today, I'd like to talk about Hermes. Hermes is a project I was setting up, I think, three or four years ago. So we use it in the OpenSUSE build service,
01:26
because there are problems I'd like to talk about in the first part of this talk. The second part of the talk will be the solution, which is called Hermes in this case.
01:42
In the third part, I give you some technical details about the system. And in the fourth part, you are invited to ask questions if you have some. So what's the problem? The problem, I'm talking about problems with emails or notifications at all.
02:02
So in the beginning, most of you probably remember that. You said to people, send me a quick email and I can sort it out and do stuff. You got important information with email. Email was fun and was a new thing.
02:21
And your work organization used a lot of email over the time and it got very, very efficient. And then machines started to send email. With machines, I mean stuff like build systems, like bug trackers, like maybe feature trackers,
02:42
and stuff like that. And they send out emails to you if your package doesn't build. It doesn't build on this and that platform. It doesn't build for this distribution, which is three years old, and stuff like that. So you end up with a lot of emails.
03:00
And all these emails look similar. You got too much of them. You're not interested in stuff like this one week ago. So with this machine sending email, everything just turns upside down. You got tons of emails. You lose important information because if you
03:20
have 1,500 emails in your inbox, you're not able to sort out which are really important and which not. It's not fun anymore. And yeah, your work organization probably breaks. So what we were thinking about was the possibility to get back control.
03:43
So we set up Hermes with the intention to give the decision back to you which information you want and which you don't want. So don't let the machine, the system, the build system,
04:00
the system send email to you. But you go to Hermes and say, I'm interested in this particular message from the build system. So it's not any more longer that you got 10 minutes from the build system because one of your package fails. It's because it fails in 10 different distributions
04:22
while you are only interested in one at this moment. So with Hermes, you can configure very detailed which informations you want to have. So that works in the way that the client system in the example of the build system doesn't deliver the message directly
04:44
to you as a user. But it delivers its notification of an event to the Hermes system. And you configure and decide in Hermes which notification
05:01
you are interested in. And it's a conscious decision, of course. So you don't get flooded with unwanted stuff, but you decide consciously what you want to have. The messages are, of course, kept in a system which stores it securely, and it's predictable and visible which messages were sent out.
05:27
And also the message itself is configurable by the Hermes admin and yourself. So you can decide how the message should look like because the style of the message is no longer part
05:44
of the implementation of the build system, which simply dumps a text file to send me. But it's part of the Hermes system where you have a configuration tool to configure it
06:01
as you need it. We have various deliveries. Also I'm talking here about mail a lot. We have other deliveries such as RSS feeds or Jabber or Twittering, which is interesting because what I set
06:20
up recently was that Opens with a Build service now fills a Twitter channel with information about package updates in the factory project. So of course, not every individual sets up a Twitter thing that's more a general approach to set up
06:42
interesting sources of information for people. What also is possible is an HTTP post that gives the possibility to send information to other systems, like on a push approach. Whenever a package, for example, changes
07:02
in the Opens with a Build service, we can do an HTTP post to another system, which can do stuff with the information. So this is something that goes away from a personal messaging approach. It's more of an inter-disciplinary communication
07:20
style. There's a web client where you can read messages. And there are tons of other possibilities which could be implemented. The idea is that every user can pick for himself in which way a message should be delivered.
07:41
So it's up to the user to decide if he wants package build failure messages, for example, as RSS feeds or as a Jabber notification. Another feature are digests. This is an example mail that is sent out by Hamas for,
08:08
I think it's a submit request in the Build service. So the actual information is this part.
08:21
This is just a feature. And for some messages, you will get a lot of these messages which just have a little bit of information. And you get them every two minutes. So what digesting means is you can combine them. You tell Hamas, please don't send this kind of message
08:42
every two minutes to me. But collect them and send an hourly digest, for example. And then all the interesting parts are collected together, combined with a header and footer. And then it's sent out on a timely basis.
09:01
So we can send the digests once a minute, hourly, daily, or weekly. Let's come to some technical details. Hamas consists of four parts at the moment,
09:23
four important parts. This is Terminator. It's a web server-based little script that receives a notification from the clients, for example, a build system.
09:40
The generator, which connects the incoming notifications and the receivers. The worker, which actually sends out messages and starts up the configuration and uses a web application. So why did we split that up? We started over with one big tool that did most of the stuff.
10:04
It turned out that, for example, the actual sending of the messages takes too long, takes time, as sometimes there were issues, stuff like that. So it's better if it runs isolated. The same is for the web application.
10:21
This is a Ruby on Rails app, which is better isolated. And in the future, we could, for example, combine Terminator and Starship again. But it's really important. As a data storage,
10:41
we use a MySQL database at the moment. There's a pretty optimized schema over time. Of course, as most of you might have noticed, you have to pay attention to the database schema if you want performance and reliability. So this is most of the,
11:03
I think most of the attention bent into the database schema. All the applications work on the database. It's pretty performant. Let's take a deeper look into each of them.
11:22
The Terminator thing is a CGI application-based script. It runs on Lighty or Apache. I think we run it in Lighty at the moment on the build service. The client interface is nothing but an URL
11:41
where the clients, for example, the build service, can post a notification, and the notification is basically just a string. So one string identifies the kind of information, the kind of notification that is happening. For example, package build fail.
12:02
This is a complete notification. And because you need more information about this event, for example, the package name, the platform, the architecture, what else, this can come in a... Well, this comes as key value pairs
12:22
as much as you want. And the system works that way, that you don't have to set up notification types or stuff. You just send them once, and then it's there. And after that, since your client often, or frequently sends the same notification type,
12:44
they come in more and more messages from the same type. So just send one, and it's there. Later, there's an admin interface where stuff can be administered. The notification type, the string,
13:00
identifies the only required parameter. Together with Selminator comes an admin interface where you can see which notification types were set up by just sending them in, and also, as an admin, can administer the text template
13:21
that is sent out for mail and for all the other things. This is publicly available under the Notify OpenSUSE.org. But you have to have an admin account, of course. The generator is just a password
13:41
that connects incoming notification to the subscriptions. Subscriptions are... People have to subscribe to the notifications. So you have to say, I'm interested in package beds, or I'm interested in feature changes, or I'm interested in whatever.
14:02
This is called a subscription, so the connection between a message type or a notification type and the user as a subscription. And with the subscription, you also state in which way you want to receive the message and when, if you want it immediately, or if it should come as a digest thing.
14:31
The worker is a person that actually sends all the messages. It does mail sending, connects to sent mail, sends it out, or does the RSS stuff
14:43
and twittering, these kind of things. Jabber. It runs in an infinite loop on the server with, of course, this break so that it doesn't cause too much load on the machine.
15:01
And whenever something gets generated from the generator, it's sent out. Starsep is the Rovian Rails application. It's a harmless user interface for users where you maintain your subscriptions, where you can read actual messages,
15:22
where you can configure RSS feeds you want to see and you want to fill with your messages. It has a user mode, which is pretty simple. The setup is an abstraction layer where people just can say, I'm interested in feature changes, for example.
15:41
And in the background, the subscription with subscription filters and some more complex stuff is set up for the people. And if you want to go one step beyond and look into the details, you can go to the expert interface and set up stuff more in detail.
16:05
Again, an overview diagram. Here's Alminator, which is just a server-based web application. The client system posts the raw notifications in there. They are stored in the notifications and parameters
16:23
tables in the database. The generator frequently checks if there are new raw notifications and combines it with the subscriptions and the filters and generates the messages that should go out to the people
16:45
based on the settings here and the subscriptions. The messages are stored in this part of the database and the worker all the time checks if there is something in and sends it out as mail or RSS. And Starship is editing subscriptions and the filters
17:05
and if you can read messages and view stuff and so on. So far, how much runs in production for our OpenSUSE project, for OBS and OpenFate.
17:24
Currently, we are scaling up and connecting more and more stuff to it since we have now fixed all the stuff that comes in the beginning. So when you start over with such a project you run into a lot of tiny problems.
17:43
Currently, we are processing more than 50,000 notifications a day. Of course, there is stuff like clean up scripts in the background which, again, clean up the database and stuff like that.
18:02
There is a project page in the OpenSUSE wiki if you are interested. And it's free software, of course. It's independent from the OpenSUSE build service so you can run Hermes for any other purpose
18:20
than the OpenSUSE build service. It's not connected, it's not using the build service code base or something so it's just an independent project. But of course, it's developed along the build service so it's pretty part of that.
18:46
So, I'm finished. Probably we have plenty of time left, right? Are there any questions from your side? You said client sends a message to the Terminator?
19:03
Yep. So there is some kind of agent in the client or is this an email actually? No, it's just an HTTP post. So you need an HTTP connection between the client and the Terminator. And, of course, you have to make sure that you do your network configuration right
19:25
because otherwise, if you, for example, open up your Terminator to the internet you're, of course, the perfect spammer because people could send notifications and if everything is set up accordingly, Hermes would send out messages.
19:42
So, you have to be careful with that. So, for example, for the open source stuff we have it configured that way that only internal network can reach the Terminator. So what exactly needs to be installed in the client systems? On the client systems, you can either use a Perl script
20:04
which provides another Perl script, a Perl module that provides one function to send out a notification. I think it takes as parameters a notification name or a string and a hash reference which contains the parameters.
20:26
So if you use other languages, you would have to implement an HTTP post which is easy for most systems, I guess. Other questions?
20:46
Right, then thank you very much, Chopra. Thank you very much, Lars.