FOSDEM 2009: OpenAMQ
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 70 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/39530 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 200933 / 70
3
4
6
16
17
18
19
20
21
22
23
24
25
26
27
29
30
31
32
33
35
40
41
44
46
47
48
50
51
54
55
57
58
59
60
62
65
67
68
69
70
00:00
Boom (sailing)Service (economics)Sega Enterprises Ltd.State of matterWebsiteCore dumpCASE <Informatik>TelecommunicationInternetworkingClosed setSinc functionComputer architectureMultiplication signInformationJSONXMLLecture/Conference
00:49
Core dumpComplex (psychology)Open sourceProjective planeInformationAreaComputer scienceCommunications protocolDatabaseLevel (video gaming)Enterprise architectureParticle systemLecture/Conference
01:28
Open sourceEnterprise architectureInformation technology consultingWeb serviceCartesian coordinate systemCommunications protocolProcess (computing)Open setLevel (video gaming)Generic programmingSoftware developerWeightComputer architectureLecture/Conference
02:18
Software developerStandard deviationPattern languageFormal languageCodeMeta elementCondition numberClient (computing)Factory (trading post)Message passingAuthorizationLine (geometry)State of matterGeneric programmingServer (computing)Communications protocolPhysical systemPortable communications deviceOrder (biology)ImplementationSoftwareSemiconductor memorySinc functionSocial classArithmetic meanTheory2 (number)Observational studyMultiplication signProjective planeSingle-precision floating-point formatWebsiteWritingLevel (video gaming)Open sourceProduct (business)MereologyCartesian coordinate systemRight angleLibrary (computing)Complex (psychology)Cycle (graph theory)AverageSurjective functionMetaprogrammierungFlow separationComputer programmingCategory of beingReal numberQueue (abstract data type)Computer configurationUltraviolet photoelectron spectroscopyMultiplicationEnterprise architectureOpen setPersonal digital assistantSound effectSystem callComputer clusterPrisoner's dilemmaFile formatForm (programming)LeakPoint (geometry)BuildingSource codeTotal S.A.Lecture/Conference
10:37
Scaling (geometry)Pattern languageMultiplication signRevision controlData structureObject (grammar)Domain nameInformation securityCartesian coordinate systemDemoscenePartition (number theory)MappingCommunications protocolImplementationServer (computing)Default (computer science)Wave packetSurjective functionAddress space2 (number)Open setUniform resource locatorClient (computing)Business objectLine (geometry)CodeEmailClosed setWeb browserType theoryProxy serverComputer programmingWordFactory (trading post)Poisson-KlammerWhiteboardContent (media)InformationDigitizingService (economics)Point (geometry)Real numberEstimatorEqualiser (mathematics)Exception handlingSpectrum (functional analysis)Cue sportsDisk read-and-write headComputer architectureScripting languageOperator (mathematics)Programming languageDependent and independent variablesLecture/Conference
15:47
Dressing (medical)Programmer (hardware)Execution unitMathematicsMessage passingSession Initiation ProtocolComa BerenicesImplementationDirection (geometry)MereologyFlow separation2 (number)TelecommunicationLine (geometry)Server (computing)Service (economics)SpacetimeInstance (computer science)Matching (graph theory)Different (Kate Ryan album)Electronic mailing listQueue (abstract data type)Router (computing)Centralizer and normalizerStatement (computer science)Point (geometry)MappingSemantics (computer science)Right angleInternetworkingOpen sourceInterprozesskommunikationSoftwareState of matterClient (computing)Physical systemPattern languageInternettelefonieCartesian coordinate systemWeb 2.0Product (business)InformationContent (media)Goodness of fitKeyboard shortcutProxy serverSubsetService-oriented architectureDynamical systemCodeKey (cryptography)Surjective functionDebuggerCompilerStability theoryAddress spaceOpen setOrder (biology)Lecture/Conference
23:20
Server (computing)Multiplication signWebsiteFunctional (mathematics)Web 2.0LengthProduct (business)Coma BerenicesHard disk driveLine (geometry)MultiplicationMedical imagingCompilation albumWindowSoftwarePrice indexVirtual machineSource codePhysical systemComputer programmingDemo (music)Library (computing)CodeTerm (mathematics)MetaprogrammierungMeta elementBitLecture/Conference
26:24
Pattern languageQueue (abstract data type)Server (computing)Object (grammar)Uniform boundedness principlePhysical systemBitTerm (mathematics)Demo (music)MikroblogMathematicsMultiplication signRight angleEmailPole (complex analysis)Message passingProgrammer (hardware)Cellular automatonGroup actionLine (geometry)Lecture/Conference
28:58
Communications protocolSocial classWeb 2.0Product (business)Formal languageProgrammer (hardware)Object (grammar)WordCodeAddress spaceContent (media)InformationGame theoryBitInternetworkingPoint (geometry)Message passingOnline chatReal-time operating systemOpen sourceLine (geometry)2 (number)outputProcess (computing)Subject indexingLoginEnterprise architectureCartesian coordinate systemMereologyTelecommunicationIntelCASE <Informatik>Service-oriented architectureQueue (abstract data type)SoftwareParticle systemEstimatorOpen setAbstractionTheoryStandard deviationLecture/Conference
33:59
View (database)Standard deviationLatent heatSelf-organizationLimit (category theory)Lattice (order)Point (geometry)Software developerWordSelectivity (electronic)Process (computing)SoftwareCategory of beingSystem call1 (number)Endliche ModelltheorieSign (mathematics)Chemical equationMereologyDifferent (Kate Ryan album)Product (business)Goodness of fitData miningMathematicsOpen setWikiLecture/Conference
36:32
Software developerRegular graphRow (database)Enterprise architecture2 (number)Message passingStapeldateiInternetworkingCartesian coordinate systemHecke operatorLibrary (computing)Service-oriented architectureInformation managementModal logicReal numberEstimatorVideo gameVolume (thermodynamics)1 (number)Projective planeRight angleGoodness of fitProcess (computing)MereologyOrder (biology)Office suiteLevel (video gaming)Bit rateDifferent (Kate Ryan album)Multiplication signStaff (military)Reflection (mathematics)Computer configurationSoftwareException handlingLecture/Conference
40:15
Service-oriented architectureHand fanBackupCore dumpVirtual machineOpen setInclusion mapEnterprise architectureStatement (computer science)Selectivity (electronic)Pattern languagePeer-to-peerMultiplication signInternet service providerOpen sourceMessage passingJava appletStandard deviation1 (number)Particle system2 (number)Product (business)Control flowService (economics)SoftwareMixed realityDifferent (Kate Ryan album)Level (video gaming)OracleImplementationCASE <Informatik>MiniDiscProcess (computing)Source codeLecture/Conference
45:55
Virtual machineOpen sourceSoftwareData centerOpen setCartesian coordinate systemSoftware frameworkCodeMeta elementParameter (computer programming)Endliche ModelltheorieElectric generatorLimit (category theory)MiniDiscDifferent (Kate Ryan album)Power (physics)BuildingBitSpeech synthesisState of matterModeling languageMetreFormal languageInformation securitySelf-organizationCommunications protocolInheritance (object-oriented programming)Coma BerenicesSocial classSource codePoint (geometry)Integrated development environmentCausalityLecture/Conference
49:25
XML
Transcript: English(auto-generated)
00:07
And someone came up with what SOA actually means, so there was state of the art, one option, Sir Baner's Oxley Act, State of Alaska, you betcha, Sextual Ovidragh Bahr Andoning and Sega of America.
00:23
I mean, who here knows what SOA actually means? Take a hand up if you know this. Right, I mean, a few, and this should be really well known because it's the internet, I mean, you know, service oriented architecture, you put stuff somewhere, you find it, it's like you want to buy a pizza, so you call someone and say, where's the best pizza shop?
00:41
And someone says, I'll put you through, and they put you through and a pizza arrives. Except it's all digital, so it should be really easy to do, because there's no pizza, it's just like, you know, a package of data, and pizza shops can come and go anytime. And since it's all about information and communication, it's a real shame that messaging, which is the core of this, is so poorly understood and so poorly made.
01:04
It's really, really expensive in most cases, it's mostly proprietary and closed and locked down. There are some open source projects and mostly they're kind of, blech. And there are some protocols and they're kind of, blech. And it really is like the last big, boring, complex and painful area of computer science.
01:22
I think we solved databases about 10 years ago, and now everyone can do SQL, it's a child's play, no problem. So messaging is in that stage of moving from enterprise technology to something that anyone can use. And today, if you can't connect pieces together across the internet, you're an incompetent developer, you can't actually make big applications.
01:41
This is what our job is. It's connecting pieces in interesting ways. So I think big companies actually like it to be complex and painful and boring. It's actually in their interest. It keeps away competition, it lets you lock in customers, it lets you charge profits,
02:01
consultants love it, you know, big fat books on web services architecture, and you have 55 protocols to learn, and they're all complex and they're all really huge, and of course it's expensive. But open source doesn't work that way. Open source works by making things simple and open and cheap and reusable and generic and, you know, everywhere.
02:20
It doesn't work by making things complicated. So this is the thing, how do you make things simple? Simplicity is very hard. It's a common idea people have that you start simple and you end up complex. It's actually all the way around. You start complex and if you're very lucky, after years of work, you end up very simple.
02:40
And look at SQL, it's a good example. You know, SQL systems from the 90s were big, complex. You have Oracle, which is a nightmare. Today it's SQLib. You know, it's a plug-in library which you just call, it just works. That's what technology should do. It takes a long time though. So messaging is something that every single project invents.
03:02
And in my company we do custom projects for a long time and we find ourselves every time reinventing messaging. There's a factory receiving orders from an SAP system. Well, there's some software that exports orders from SAP and then sends to the factory and then processes them and sends out confirmations. We take messaging all the time.
03:21
And every team does this, I think, and we all do it badly. That's the funny thing. It's a very easy problem to try to explain. The question is very simple. It's I want piece A to talk to piece B. And piece C and piece D, that's easy. But actually doing it is very, very hard. Doing it properly is very hard. And it's hard because you've got to get the addressing right.
03:42
You know, where is piece A? Where is piece B? What if they move? What are they called? You've got to get the queuing right. What if piece B is gone temporarily? What happens to its messages? You know, what if I have several piece Bs? How do they share the work? These things are subtle. And the answers take time to develop. Reliability, transport systems.
04:01
What protocol do I use? Is it TCP? Is it SETP? Is it multicast? You know, what do I use to connect the pieces? The lower level. So the right answers are hard to get. And in fact, until recently, no one had the right answers, that's my theory. Lots of wrong answers. Big expensive wrong answers. Fantastic.
04:22
So a few years ago, I think because people got really fed up by this, and by people I mean large companies, users of middleware messaging, actually began to push develop a new standard called AMQP. And AMQP is very interesting. It's worth studying. I think it's the first open standard
04:41
that actually tries to address the generic problem of messaging. And it's a very generic solution. You could use this for anything almost. It has a general answer to addressing, a general answer to queuing, a general answer to naming pieces. It's portable. It has no hang-ups about what language to use or what data format. Its blobs move from A to B to C.
05:04
I guess I was the first author of AMQP, so I kind of love it. But I think it was really an interesting protocol. And it's been adopted by several companies. There are, I think, five or six implementations now. And they're being used in anger, in large deployments. One of our customers is Dow Jones Company,
05:21
which now moved their industrial average onto AMQP backbone, which is quite impressive. They're doing something like, I think it's 100,000 messages a second, and that's constant the whole time, full time, on and on and on. So they pull in data from Reuters, from NASDAQ, from Chicago Options Exchange,
05:40
and it goes through a server which filters it, selects it, and then it gets published out to different clients, which then bring it down and it computes, amongst other things, these averages. So you have a protocol designed by users in small IT firms, which is very good. It makes it quite reasonably nice.
06:01
The website is amqp.org. Website's kind of disappointing, as you'll see. Now, OpenAMQ, which is what this talk is gonna be about, but that's a very small part of the whole thing, is our AMQP server and client. We began making this together with AMQP about four years ago.
06:22
It's, I would say, industrial quality software, very serious stuff. It's easy to use, but it's also reasonably painful to build a large build system, multi-threaded, designed for big use. But yeah, download, build, compile, run, fairly straightforward. It's fast. Could be much faster, but AMQP has issues there.
06:43
And it's reliable. Like I say, it runs and runs and runs, and we spent something like three or four years just making that work. Very hard to make reliable software. You know, you get memory leaks, you get little race conditions. If it's multi-threading, it's horribly difficult, but we got that. We have a whole toolkit, in fact. We had this crazy idea of writing it in C.
07:02
C is the worst possible language for serious programming. It's really a terrible language. I love it, in fact, but it's really terrible. It's very simple. It's not like Perl. Perl, you learn Perl, you write your program, you forget Perl, and you go through this learn and write and forget cycle every time, and it's really painful. C, you remember C. It's very simple. But when you write C, every line of code
07:22
is really painful to write, to write huge amounts of code. I think that the size of OpenAMQ in lines of C code is approaching one million. And that's a lot of code for a small team to maintain. So we use a toolkit, which is a meta-programming toolkit, which is really funky. Meta-programming means that you actually write code
07:41
that then writes code. And what we do is we write code that writes code that writes code that writes code that writes code that writes code. It gets really crazy. But it does actually produce a lot of code very cheaply. The actual, so a million lines of code for OpenAMQ, the meta code is about 150,000 lines of code. And of that, I think only about 70,000
08:03
is really written by hand, and half is generated. And then that produces all the C code. Meta-programming is a way to write very, very large systems which have a kind of a, you know, guaranteed level of reliability. When you generate code, you know what's in there. There's still handmade code in there,
08:21
so there's still issues, of course. But they can be fixed. So that's on www.openamq.org. Now, we built AMQP for JP Morgan. We then built OpenAMQ for several clients. We delivered that, and then we looked at this for a while and then we said, this is still too complex. People aren't adopting it. It's look at amqp.org, look at OpenAMQ, try it.
08:44
You'll find it too complex. Try to write a client that speaks AMQP. It's a protocol. It's written, it's documented. It takes months. That's not good. I don't like that. For me, the target audience for technology and products
09:00
isn't the guy sitting in a bank with a year and a half to write a product, you know, to write an application. That's not fun. My target audience are people like me, open source developers who have no time, who want to download, try it, and it works the same day, or else they give up. And for me, if you don't get AMQP and stuff like it, and it's way of doing messaging into the hands of,
09:23
you know, developers without, you know, an enterprise, you know, financial system to back them, which is their own time and effort, then it won't work. And we will see the domination of messaging by proprietary products and by really second class answers. So I've been working with some guys since August
09:42
on making AMQP simpler, simpler, simpler, simpler, which is quite a fun project. There are other problems with AMQP. It has some flaws, but we were still learning when we made this, you know, it's not perfect. We discovered something called REST. Now REST is fun. REST is something which has become very fashionable
10:00
in the last year or so. But in fact, it's the underpinning of HTTP. It's a very straightforward way of working with resources. What REST does is it says, look, we have a server with stuff and clients, and the clients can use create, read, update, delete, patterns to work with the server resources. It's very simple.
10:21
It's not complicated. There is, the state is very clear. You know where things are at any point. You can redo, if you're not sure, you can try again. You delete, okay, delete again, no problem. So it's a very robust protocol, whereas most protocols are very fragile. If you get it wrong, it breaks. REST is very tough.
10:41
It's a pattern which works over large scales and which is easy to use by people who don't have much time for the fine stuff. I quote from a user who was comparing REST to SOAP and XML RPC, and he said, moving to pure HTTP allowed us to throw away the code for the XML RPC service in brackets.
11:01
I love throwing away old code. And so I was saying, look, let's try and make AMQP work over HTTP. It's obvious. I mean, the patterns don't quite fit, but HTTP has, everyone can program HTTP. I go in Perl, I say, I pull down WP, and I say, UA request, click response,
11:22
and I get something, and I do it in five minutes. So, okay, this was the challenge, getting AMQP working with HTTP. It took us some time. It's now there. It's called REST-MS.
11:41
And what this is, it's a very clean REST, RESTful implementation of a messaging protocol which actually maps purely onto AMQP. So your clients, your applications speak REST-MS, which is HTTP with put, get, certain resources, which I'll explain in a second. And it's just HTTP.
12:01
It's just a web client. Could be in JavaScript, could be in Perl, could be in Python, Ruby, whatever, and speaks to a server which then maps that onto AMQP and then speaks to an AMQP server and back and forth. So it's like a proxy between AMQP and HTTP. What this means is that you can basically get from bare metal, well, bare programming language,
12:22
metal if you like, onto working messaging application in a matter of a few hours. Less. That's on www.restms.org. That's an open protocol spec which anyone can contribute to.
12:42
You know, some things with AMQP I don't like is that the protocol is managed by a small group of people. It's very exclusive. If you wanna contribute to it, you can't. Forget it. They wanna accept contributions. I don't even know what's happening in version 1.0. It's happening behind the scenes. This is kind of annoying. If it's an open protocol, it should be open to anyone to contribute. So with REST that's the way it is.
13:01
If you wanna have an opinion on it, come with fixes, it's all there, it's all online. I was gonna show you an example but very small. Basically it's six lines. I'll read it at you because who does not know HTTP in a modest sense?
13:22
Okay, no hands, that's good. So it starts and it says post. You know what post is, right? So it says post slash restms slash domain slash default. This is saying speak to my default domain object and make it create something. So the default domain is like the factory for other objects.
13:41
Content type, application slash restms plus XML. That's just saying I'm gonna send you an XML document. Hang on a second. Then it gives a slug. A slug is like a name. It's a slug. Two points fortune, that's the header. Three lines of header. And then comes in an XML document, woof, which has brackets, restms.
14:03
Feed type equals service, close brackets, restms. That's it. So it's a little XML document being sent to a factory which is a domain. I'll explain these words in a second. This tells an object on the server to create a new object. So the server does the creation of objects
14:20
and then it comes back with an HTTP 201 created with the location which is the address of the object, which is the URI. And now the client application takes a URI, does stuff with it. When it's finished, it says delete on that object. So it can do read, update, delete on that object which has got created. So the whole structure is self-navigating,
14:43
created dynamically, and navigated through URIs. This is very, very, very nice. It means that you can completely partition work between applications without any kind of notion of security or training. If you know your URI, you can access it. If you don't, you can't access it. You can have public and private resources
15:03
and clients don't have to invent anything. Clients get information from the server, they use it and they give it away. Okay, now I'll explain very briefly MQP and restms as architectures, how they work. It's actually quite simple. You have a board and you have chalk.
15:25
So the problem is you have applications here that want some kind of a service. They want to get pizza. And you've got restaurants here that have pizza. They all have pizza.
15:42
It's digital pizza, so there's no real pizza. And there's someone in the middle that can actually tell you where the restaurants are and how to get the pizza backwards and forwards. So you kind of say like, okay, I want a pizza.
16:00
That's like your address. And someone here says like, yes, we have pizza. And in the middle, the request and someone's statement of we have pizza get matched up and the request for pizza comes through
16:21
and it's pepperoni. And then that gets sent back a pepperoni pizza. So a few instance later, this pizza restaurant burns down in a ball of fire because the guy left the stove on too long
16:41
and his wife was shouting at him. And this one is now visible or it was visible before but didn't get paid attention to. So the thing about services is that they come and go completely randomly. Certainly as a user here, you have no idea who you're talking to. It's indirect. It's abstracted.
17:02
This is very, very important. It's easy to make architectures. Problem is, things change. And in all IT, it's change which is the big enemy. Stability is not an issue. Anyone can write, you know, a programmer, a compiler for correct code. Anyone can write systems that don't accept change. But change happens a lot. Here you can change stuff completely arbitrarily
17:21
and that's the magic of messaging. You can add and remove pieces and they find each other dynamically. They do their work and that's it. No state necessarily shared between them. And there are different patterns. For example, in this pattern, the message goes back through this central router.
17:41
But in fact, that's not necessary. You can actually do very nice peer-to-peer messaging where you use the central point for address lookup and then they talk directly. And if you know something like SIP, which is the voice over IP application that works like that, the central registrar for who's online and where they are, and then peer-to-peer communications.
18:02
And we do this in some of our products. There's one called ZeroMQ, which works like this. It's very nice. Much, much faster than AMQP. But AMQP insists on everything going through one single central point. Now, the key concepts that AMQP, I think, introduced and which work very, very well are the concepts of an addressing thing here,
18:22
which we call an exchange, and queues, which connect to exchanges through bindings. These are queues.
18:43
I don't know how many Es here. Okay, that sounds quite right. So queue binding exchange lets you create a queue that takes care of the asynchronous aspect. If there's no pizza restaurants for a second, it'll wait in the queue. When one comes, it'll get its work. So queuing is essential to any kind of a robust system.
19:01
And the exchange lets you then have a fixed point to talk to, but which then will handle the things that come and go. So you talk to a fixed point. You say, look, I want these and these messages. When they arrive, you get them in a queue. This make sense? Okay, this certainly works very well.
19:21
But in AMQP, there's a few gotchas in there. For example, we have different kinds of queues. We have queues which are shared queues where you basically say, look, all my pizza orders will come in this queue. I get them one by one. I share them with different pizza restaurants. They're big, important queues. We also have little private queues, like I am subscribing to the gold price.
19:42
I want these prices in some little private queue for me. They're in the same space, which is kind of messy. There are a few other problems with that as well. But, okay. Rest MS works in a very similar fashion, but we use different names just to be annoying.
20:02
We call these feeds. We call these pipes. And we call these joins. So a pipe is like a Unix pipe. You create pipes. You consume from pipes.
20:20
It's a very simple metaphor. They have names, but they're server-generated names. You say, I want a pipe. Server gives you a URI, as we saw in the RESTful example. You then say get, which gives you a message. You say delete, which deletes the pipe. You get a message, message content, so you can access the pipe and what it has. The feed is where you send stuff.
20:42
So a feed would be for publishing information, for sending requests for pizza, anything. And then you connect the pipes to feeds. It's a very simple metaphor and it works. And it actually maps onto AMQP in a kind of a nice way. It's not quite one-to-one, because in some of the feeds are exchanges and some actually are queues.
21:01
And that's where AMQP got it wrong, is that it actually mixes things which have the same basic semantics in two different places, separate semantics. Okay. Enough of the lecture on that. It gets very detailed. We made an implementation of RESTMS called Xire. So if you were online, I would say,
21:21
look at xire.com now, but you can't. But I'm sure you will when you go home. Xire.com has a... So Xire is the messaging server. It's open source, of course. We began this last year together with RESTMS. And it's basically a proxy
21:43
for... I'll show you how it works. Typically you have an AMQP network with a big broker in the middle and applications hanging off it all around,
22:04
doing heavy stuff. And then you have an internet network on the outside, public internet network. You have clients here that want to access something in that network. Xire sits here. It speaks AMQP.
22:27
And here speaks HTTP. You know, post, get, delete.
22:46
So basically you can use Xire in several ways. You can use it to create a web front end for an existing AMQP application. Which is unlikely to interest anyone here because I doubt any of you do that. More interestingly is just...
23:02
completely ignore this part and just use it as a messaging server. And just pretend it doesn't know. There's an AMQP engine at the back. Not relevant. It's basically a very simple, accessible messaging system which speaks HTTP. That's all.
23:25
I think the... Again, it's always to do with the cost. The cost of learning AMQP, writing an AMQP application, it's weeks and months. This is a big improvement over previous systems already, which used to be six months to a year.
23:43
Getting, you know, JMS. Reasonably good. Still very, very complex. Doesn't do much more, basically. AMQP is good, but it's still too complex. In REST-MS, you pick it up. You should be up and running in a few hours.
24:02
If you're using one of the libraries that we have, less time, a few minutes. And that's really the goal. I mean, the goal is to get this to be really, really accessible, really easy to use, and still deliver the same functionality. We lose a few things here and there. Some things we chop off. Not interesting.
24:21
Maybe they come back later when we actually need them. Xayah is actually built on something called X5, which is a redesign of an old thing that we made years ago called Shitami, which is a web server.
24:40
A very popular, tough little web server. I once saw on Slashdot a guy who had a Windows 95 machine with 26 hard drives, and he had this site running on Slashdot. It was being Slashdotted and running Shitami, and it was just taking it. Very tough. This was one of our early experiments in multithreading in code. It's now being rewritten to use our new engine,
25:03
and that's doing the HTTP work for Xayah, in fact. So we'll also make a new web server, which has a kind of a side product. To give you some examples of how big, this is actually about 3.5k lines of code. Meta lines of code, if you like.
25:21
Xayah is about the same. They're tiny in terms of actual code work. The meta programming is very, very efficient. So then we were saying, well, that's fine.
25:40
We have this software. It's a little bit complex to install. We don't yet have nice binary packaging. My friend Benjamin has been working for like six months. Where's Benjamin? He's not there. To try and get Debian packaging working. It's very difficult. This is all FUDs work, and it takes time and stuff. So we still deliver source code. It's annoying for many people.
26:00
Especially on Windows, where with Microsoft compilers, I mean, they're just so bad. So we made a live site. This is fun. It's called LiveXayah.com. And again, if you had Wi-Fi, you could go and try it right away,
26:21
but you can't. So just imagine it. Basically, it's a running RESTMS server. It's running online. You can access it. It's an open server. You can use it to publish stuff, receive stuff. And I had some demo programs, which you can see if you go to zara.com. You can look at them and try them. It's a seven-line Perl script, basically.
26:41
And it's doing microblogging. So you publish to a feed with your name and a comment, and you can receive from a feed. And you can basically just do microblogging like that. And it's really, really simple to do. So you're probably asking by now, what's actually different here? We know a lot of stuff. We have RSS. We have Atom, Pub. What's different?
27:01
What am I getting if I try this out? So I'll try some sales. I don't usually do sales. I'm usually a programmer. But OK. I think the biggest thing is that stuff's happening here, interesting stuff, useful stuff happening on the server on your behalf. The most interesting is queuing.
27:21
Now, if you look at RSS, RSS basically says, here's a feed. And every now and then, you go and look at your feed again. You poll, and you say, where's my feed? And it's very slow. And you can maybe get 10 or 15 or 20 items. Then the things hang. And it's horribly slow. For a little bunch of data, it's incredible. And it's all polling.
27:40
So you have to poll, poll, poll, poll. And you're asking the server, has something changed? Has something changed? Has something changed? You couldn't imagine a less efficient way of doing messaging. It's the worst possible design. AtomPub is better. It's definitely got a better pattern. It's very RESTful. If you look at AtomPub, the spec, how they work,
28:00
it's very nice in terms of REST. But it's still polling. You're still saying, here's my feed. Get my feed. Look at my feed. OK, fine. Get my feed. Look at my feed. So new messages, you have to go and ask them, when do they arrive? This is a really bad way of making any kind of system. So systems should be asynchronous. You should be event-driven. That means that it arrived. Thanks. And most of the time, you're waiting on stuff.
28:22
So you say, I wait. When it arrives, I continue. What we do in REST MS is we have what's called a long poll. A long poll says, get a message. It's not there yet. When it arrives, I get it. So the get action, the get method,
28:41
actually can be an asynchronous get. It's kind of weird. The URI exists before the object exists. And it works very nicely. So you can have queues which are sitting on the server, waiting for you, private queues. And then when you're at your leisure, you collect your messages. It's really like a mailbox, a little bit like email,
29:00
but much simpler than email, much, much simpler. Addressing. So again, protocols like AtomPub are very nice, but there's no abstraction. There's no addressing. You publish to a feed. You read from a feed. Well, it's better than publishing directly, point to point.
29:23
But it's still very tightly coupled. With REST MS, you can say, look, I have a pipe, and my pipe collects data from these and these and these feeds with these conditions, clack, clack, clack, clack, clack, clack. I get my stuff in my pipe, and I read it when I want.
29:42
So the pipe is actually doing a little bit like searching on feeds of data in real time. And whatever is matching, it picks up and puts aside, and then you can pick up in your application. So you can imagine new ways of doing information processing. Let's say you want to monitor IRC channels on Freenet, which is, by the way, not allowed.
30:01
I tried it, and they're like, you can't do that. Okay, fine, whatever. And you want to see when they mention your product or your name. You know, you have all these channels talking. Maybe they're talking about you. This is a great paranoid tool. So you can search, of course. You can export IRC logs. You can index them, and you can search them. That's useless. You want to get in on the chat. You want to come in when they're talking about you and say,
30:21
yeah, that's true, that's great. He's a great guy. I know him personally. Or, yeah, that's a great product. Or, no, no, it's all lies. It's all lies. But you want to be there in real time. Information when it's old is useless. This is really the way it goes. Information, the value decays really quickly. So you want to be able to monitor the whole of Freenet in real time. Well, you take, you subscribe to all of the channels.
30:43
Don't do this. This is why they're not allowed, but this is what you would do if it was allowed. Subscribe to all the channels. Publish all of the content of the channels onto a feed on LiveXire.com. Create a pipe and say on this pipe, I want to match on these and these and these words.
31:01
Read your pipe. That's it. That's simple. Conclusions. Messaging can be fun, and it should be fun. It's communication. It's what we like doing best of all. So the Internet's all about is communicating, making pieces work together. The fact that it's really boring and complex and painful
31:23
is because it's been done by big companies who are basically incompetent or greedy or stupid or all three. We have new open protocols that change the game. They really change the game. The messaging has been dominated by products, first of all.
31:40
APIs, secondly. And protocols, really, it's a very recent phenomenon. APIs are useless. That's ODBC. You know, it didn't get us anywhere. You want standard protocols where anyone can compete. You want open source products anyone can improve. You want competition. You want freedom from lock-in. You want the programmers actually fighting to make a better product.
32:04
So, for example, with OpenAMQ and AMQP, you know, we find it was too slow, so we have some guys in Slovakia, friends of ours, fastMQ, and they make a product called ZeroMQ,
32:21
which is basically doing a lot of this work with no broker, and they're hitting 4 million messages a second with latencies of about 10 microseconds, just in software. Really extraordinary. This is competition. People saying, I can do better. And it's all open source, so you get the input from, you know, the input from Intel, from other people helping us with that.
32:44
AMQP is good. It's still too complex for me. It's failed in its mission of being simple and accessible. It's enterprise technology, which is really not good. All that means is that it's a problem. You want it to be web technology for me.
33:00
So, my advice is to have a look at RESTMS. Try it out. If you're a Perl programmer, you're lucky because I made Perl classes which wrap up the basic objects in Perl. If you're a Python or a Ruby or a Lua programmer, and you feel like it, then, you know, rewrite my class, my Perl class,
33:22
and contribute as a class in other languages. Go for it. Try it out. Tell us what you think. It should, in theory, be simpler to use an XML RPC. Much, much simpler than SOAP. And it should be very close to production ready. We're talking about code that's been, you know, there's,
33:41
here's three and a half K lines of code that is new. And it's running on an AMQ backbone, which has been running for years and years in production. So, it's very close to production ready, the whole thing. That's it. I'll take questions now.
34:07
It's a good question. The way I see it, so there's different points of view in this. My point of view is standards is work of the community, and that it's a matter of selection,
34:23
as in natural selection. In other words, standards, for me, should develop by an organic process where people propose small improvements and the good ones get adopted, the bad ones get run away. So, for me, the best approach to standards I've seen ever in IT is the RFC model, where there are lots and lots and lots and lots of small standards.
34:41
And interoperability is a matter of that one, that one, that one, that one, that one, and that one. You know, I implement RFC 1218, I implement RFC this and this, and they're all very formal, they're all easy to verify, because the standard is a small piece of specification. That's my view. AMQP doesn't really do that. AMQP is getting very big monolithic standard.
35:00
So, interoperability with AMQP, as it stands, is a difficult thing to even prove, let alone achieve. What I've been working on, apart from software, is a process around AMQP to encourage community-driven standards development. And RESTMS is part of that. There are a bunch of other standards that we've been developing,
35:21
small standards, which I call supporting specifications. And there's a wiki site,
35:47
Guerrilla, Guerrilla Standards Development. It's how to do it without a top-down structure, without committees, without meetings, without that. And it's based on work we did in other organizations, like Digistan, how to make good, simple, accessible open standards.
36:03
If you're doing any kind of standards work, it's fun to look at that. And that's interoperability. It's based on people trying specs, implementing them, saying it's good, it's not good. And if it's good and you get two or three companies implementing it, teams implementing it, you have interoperability. You don't interoperate with products, only with standards.
36:22
Otherwise, you get this business where, well, my product is doing this, well, I'll change mine to make work with yours. That's crap. You interoperate with written specifications. And if they're not simple and small, you can't do that, and therefore, you don't get interoperability. Next question.
36:46
Regular developers. Regular developers. Can you repeat the question? Yes. So the question was, is the messaging for regular developers or for enterprise? Roughly. Well, there's no contradiction.
37:02
The thing is that enterprises, and enterprises we mean big companies with many staff, have a very high tolerance for pain. They don't mind using really complicated technologies, which take a long time to learn. They don't mind paying a specialist full time just to learn what the heck SOFA is. I mean, they don't mind that. Little companies do mind that.
37:21
So little companies needed to be very cheap, very simple, and very effective very quickly. Actually, they're quite small, to be honest. They're well known, but they're actually, the team there is like four or five people, and they're very, very good. And they actually pushed us to make things a lot simpler, which is interesting, exceptional.
37:44
That's very unusual in large businesses. I've had a good team, but they were in fact a team that used to work at a small Internet startup called MarketWatch, which was bought, and so they're very competent people. It's unusual. Most big companies are filled with very incompetent people. I don't want to be unkind, but that's the truth.
38:01
And they don't really want, they want things to be comfortably difficult, make everyone's life in a certain way easy, big budgets, big projects, not necessary. So my goal is to really make this easy for people like me, who are basically lazy. I don't like having to be really concentrated to understand something.
38:21
I want it to be simple, to work automatically, to be foolproof. It's more fun, and it's also part of the art. If you make software that good, that it works for anybody. It's an achievement. Next question. Dieter. Right.
38:46
Yeah. So there are, the volumes that we talk about, so the volumes talk about, they go for anything from one message a second up to five million messages a second.
39:03
That's the volumes we talk about in real life use that we see. If you're doing a, if you're picking stories from the BBC, it's one per second, if you're lucky, if they're working really hard. If you are processing orders on the Chicago options exchange, it's five million per second.
39:24
So obviously there are trade-offs in different ways of doing things. This guy here aims at that level. So if you want to see how to make stuff, sorry, yeah, yeah. If you want to see how to make stuff that can process millions per second, look at ZeroMQ.
39:41
It takes a different approach. There's no broker. Basically there's a library which you plug into your application, and you have to write your applications very, very, very carefully, because if you make mistakes in applications, it slows everything down. And then the other end, I have RESTMS, which I don't think it's one a second.
40:01
I think it's probably about 100 or 500 a second. But it's very chatty, RESTMS. And REST is very chatty. It's get my message, delete my message, get my message, delete my message, one by one. Now, you can batch it. So I'm not really worried about if I have 100 a second. I can make it 10,000 per second by batching messages, which we do a lot also in MQP.
40:21
So the trick is you make it simple, you make it work. You find out where it's too slow, you make it better. You don't start by making it fantastic. You start by making it simple and making it work. And then when people say, look, it's too slow for me, make it better. MQP is somewhere here. It can do something like 100,000 to about 500,000 a second,
40:47
depending on the implementation, which is more than anyone needs outside of certain large companies.
41:04
You mean like choreography? Sure.
41:20
I would say not my problem. The thing is that we work in layers. And this layer is about shoving blobs of data around the network between peers. We're doing addressing, we're doing queuing, we're doing simple APIs, we're doing simple patterns for reliability. That's it. On top of that, you can build all kinds of services, including...
41:52
What I was going to say was it'll come. So when you have layers that work and are trusted, people will build on that.
42:00
Companies like Tipco need to always raise the bar because they're selling stuff. Open source, we don't care. We can stay quite happily at one level, make it better and better. But it'll happen. People using Tipco will say, well, why am I paying 10,000 per core per year for this and I can make it myself on top of OpenAMQ? Okay, go, make it.
42:23
At the back, a question. No, at the front. I'm looking for a messaging solution. No. And there are many, many messaging products. Many. Thousands.
42:41
And there are many open source products. The trick is, what are you actually buying? And this is the thing. We have product, we have API, we have protocol.
43:09
And again, you compare it to SQL. You can be buying Oracle. You're buying a product. Well, if you're comparing Oracle to Sybase, to Informix, you're not winning anything. You're just choosing which lock-in you're going to get.
43:21
Now, open source, it's better, but it's still a product. You're still locked in. Okay, you have APIs. APIs are better. But APIs don't actually guarantee you very much because APIs are always extended, modified and changed. ODBC, again, a good example. Now, so, you want a standard. SQL 92, this is the way a select statement works.
43:41
If you stick to that, it's going to be portable. And now suddenly you're free. And now you can take Oracle, put in MySQL, and if you stuck to that SQL statement, it'll work. So actually, when you're buying technology solutions, if you're buying products, you're just choosing a way to be locked in. Unfortunately.
44:01
APIs is better, but it's still not enough. The problem in Java is Java is all APIs. You have GMS messaging. It's an API, and every single provider is different, and they all extend different ways, so you're just choosing different kind of lock-in. So it's about standards. Standards are very important. DDS, yes, it's a protocol.
44:22
It's patented, and it's not well documented, and, yeah, okay, what can I say? I'm not a fan of DDS. I'm not a fan of patents, indeed. How long do we have still?
44:42
Did anyone get Wi-Fi working in the meantime? Did you get Wi-Fi working? No? Oh, sorry, okay. Then you're in trouble. No, I'm just joking. Brokers fail. There are different answers to that question. The one I like best is the broker shouldn't fail.
45:05
And, in fact, we put our effort into making a broker that doesn't fail, which is more fun than making a solution for when the broker does fail. So it's like, you know, I have a car. What happens if my car breaks? Well, what we'll do is we'll sell you two cars, and then we'll drive behind you with the second car all the time.
45:20
The first car breaks, you jump in the second car, you carry on. It's fantastic. This is what they do. I'm serious, okay? It's called a backup broker, and you actually have two brokers running and two big machines, and one is going vroom, that one's going vroom, and this one dies, so then you switch off the backup broker, and then you hope it all works. Nah, useless. This is what enterprise likes, but it's useless.
45:42
It's really a bad way of making reliability. Yes, machine could break down. So backup machine, possibly a good idea. Possibly two machines if you're worried about your machines, but today machines don't break down anymore. Again, machines are very reliable.
46:01
They have fail-safe power supplies, they have fail-safe disks. Okay, your data center could burn down. Okay, your building could explode. Sure, it could. It's really a matter of how paranoid you are. What we discovered in the end was that you can actually make a software completely and utterly reliable within the parameters of your application.
46:21
You can always crash it doing different things. We have stuff that runs day and night in company A, and some other guy says, look, I do this and it dies. Like, wow, that's something new he does. So software is never reliable without limit, but within the limit of a particular environment, it can be 100% reliable. At that point, it never crashes.
46:41
And then after six months or a year, the customer says, wow, it never crashes. And if it does crash, it restarts really quickly. And if you take an example like Dow Jones, they don't mind if it crashes, in fact. As long as they get the data back within one minute, as long as they get no duplicates. They don't mind if they lose data, in fact.
47:02
But they hate getting data twice. Imagine that the stock price of Microsoft is going up and suddenly goes back up. You know, that would be crazy. So a gap is fine. Data later in the wrong order or double is really bad. So companies have lots of different requirements.
47:21
There are different answers to that question, basically. And my answer is make it really reliable and, you know, try not to make it crash, try to stop it crashing.
47:42
Unfortunately, we've been doing this for something like 15 years now. Co-generations. We have our own tools for that. We have co-generators. We have one called GSL. I mean, it gets really, really complex. I don't want to talk about it too much because this is not simple. But it's a co-generator. We have frameworks built around something called XNF,
48:04
which is, again, a modeling tool. And if you look at imatics.com, you'll find some reference to this. Co-generation is unfortunately very leveraged, so it gets very abstract very quickly. And that's a bit of a shame. It means that people don't understand what we do.
48:22
In fact, when we write software, we invent new languages, meta-languages, modeling languages, which describe protocols, describe state machines, describe classes, and we generate from that. We generate the generators. It lets us work very quickly, but it also means if you look at the code, you'll be like, what the fuck? This is impossibly complicated code.
48:41
The generator code is really, really complex. And the meta code is very abstract. So it's a bit of a difficulty for participation. It does let us write very good software very quickly. It's open source, yes. All open source. All the meta code is GPL. All the generated code is, I think, BSD.
49:09
Okay, hands up who hears for the next speech and not my speech. Just to see. Okay, thanks guys.
Recommendations
Series of 70 media