Marconi
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Alternative Title |
| |
| Title of Series | ||
| Part Number | 108 | |
| Number of Parts | 119 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/20049 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Place | Berlin |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
EuroPython 2014107 / 119
1
2
9
10
11
13
15
17
22
23
24
27
28
41
44
46
49
56
78
79
80
81
84
97
98
99
101
102
104
105
107
109
110
111
112
113
116
118
119
00:00
Open setBenachrichtigungsdienstService-oriented architectureSoftware engineeringPoint cloudComputer animationLecture/Conference
00:39
BenachrichtigungsdienstRed HatArchitectureCycle (graph theory)Web serviceIndependence (probability theory)TelecommunicationMessage passingQueue (abstract data type)Task (computing)Point cloudHorizonEvent horizonStatisticsPersonal digital assistantData storage deviceVirtualizationWeb serviceProjective planeMessage passingArchaeological field surveyExterior algebraService-oriented architectureVirtual machineMultiplication signMaizePoint (geometry)Set (mathematics)Independence (probability theory)Data managementOpen sourceAreaBitStandard deviationFormal languageCASE <Informatik>Cycle (graph theory)Electric generatorPoint cloudGroup actionProduct (business)SpacetimeTask (computing)Cartesian coordinate systemOpen setRow (database)Integrated development environmentOrder (biology)Queue (abstract data type)IntServCentralizer and normalizerLecture/ConferenceComputer animation
07:40
ArchitectureAuthenticationImage warpingData storage deviceMessage passingEvent horizonMultiplicationService-oriented architectureIntegrated development environmentAuthenticationMiddlewareArithmetic meanExterior algebraCASE <Informatik>Web serviceOrder (biology)Projective planeControl flowMoment (mathematics)HorizonLevel (video gaming)Product (business)Centralizer and normalizerArchitectureTerm (mathematics)Multiplication signData storage deviceCommunications protocolQueue (abstract data type)StatisticsImage resolutionGroup actionPlug-in (computing)Cloud computingClient (computing)Serial portProgram flowchart
14:26
Data storage deviceSingle-precision floating-point formatCluster samplingIndependence (probability theory)Open sourceScale (map)Web serviceCharacteristic polynomialQueue (abstract data type)Human migrationPersonal digital assistantSummierbarkeitIndependence (probability theory)Basis <Mathematik>Data storage deviceMultiplicationReading (process)Ocean currentGene clusterHuman migrationQueue (abstract data type)Cartesian coordinate systemSemiconductor memoryWritingSingle-precision floating-point formatBitCASE <Informatik>Right angleInstance (computer science)PlanningPlug-in (computing)Stack (abstract data type)FeedbackOpen sourceView (database)AreaWeb serviceFitness functionParallel portProcess (computing)RadiusElectronic visual displayRadical (chemistry)Student's t-testSocial classGame controllerMultiplication signMoment (mathematics)Data structureOpen setDialectRow (database)Computer animation
21:12
Cartesian coordinate systemDifferent (Kate Ryan album)Level (video gaming)Service-oriented architectureBlock (periodic table)Product (business)BuildingWeb serviceQueue (abstract data type)Core dumpData storage deviceDecision theoryMessage passingINTEGRALMereologyNeuroinformatikInstance (computer science)Condition numberIntegrated development environmentExpressionComputer programmingInstallation artOpen setRow (database)ConsistencyLecture/Conference
Transcript: English(auto-generated)
00:15
Right, so we'll start with Iela Kaplan, who's gonna be talking Marconi OpenStack
00:21
queuing notification service. And Iela is a software engineer at the cloud team at Red Hat. Thank you. So thank you. So I'm Iela.
00:41
I'm both nervous and excited to be here. So I'm working for Red Hat, mostly around virtualization. I've been contributing to the OVID, mostly around the storage area, and to OpenStack to the Marconi project.
01:01
So today I want to talk to you about Marconi, which is a queuing and notification service for OpenStack. So what we're going to talk about today is why do we need a messaging service for OpenStack? What is Marconi? I want to give you a high-level overview
01:21
of what Marconi actually looks like, and we also give you some use cases about how you can actually deploy Marconi and use it in your own cloud environment. So let's talk a bit about the project's history. So the project doesn't really have a big history.
01:42
It's pretty young. It was started in January 2013 by Rackspace and Red Hat. It was incubated into OpenStack during the high-source dev cycle. It is currently production-ready, and we really hope it will get
02:02
into the next JUNA release into OpenStack. So first of all, before we start talking about Marconi, I want us to talk a bit about OpenStack. So here you can see a picture of the OpenStack services.
02:20
So OpenStack is basically a set of services that are running on distributed machines throughout your infrastructure, and they are pretty much independent to one another, but they also need to talk to one another. So currently they're using a centralized message broker
02:43
in order to do that, and what we want to do with Marconi is introduce to you an alternative to this message broker in cases where the message broker is just not good enough or not secure enough for your use case.
03:03
So as I said, we have a lot of independent services in OpenStack, and there are also a lot of messaging technologies, which means different languages, and we want to have one unified way for the services to talk to one another.
03:27
So basically we have a missing piece in OpenStack. We want to have a couple of things. So first of all, we want to have a queuing service for OpenStack. We also want to be able to have a notification as a service.
03:42
This means that we want a service to be able to publish messages to other services in OpenStack. Also, we want to have a really lightweight messaging API, which means it will allow us to integrate services using one unified API that will be really simple to use,
04:03
and it will cause no extra cost to your infrastructure while deploying it. So how are we gonna do that? You're probably wondering if I'm talking about just yet another messaging broker, because we have so many.
04:20
Why would you need just another one? And there's also a really nice joke about it. You probably know it. So I want to reassure you that you don't have to worry. In Marconi, we are not aiming to replace any existing messaging technologies.
04:41
We are aiming to use existing messaging technologies and to sit on top of them and use them. Also, Marconi is not aiming to be a task manager.
05:00
We have Celery, which is a distributed task manager, and Marconi aims to talk to Celery. It is able to talk to Celery, and we want Celery to sit on top of Marconi and use it. Also, Marconi is not a queue provisioning service. It will not allow you to install
05:20
any of your underlying technologies. You will still have to install everything yourself and deploy everything yourself. You will still use the messaging technologies you are using today, but you will use Marconi on top of these technologies. So now that we know what Marconi is not,
05:41
let's talk about what Marconi is. So Marconi is a really simple and lightweight RESTful data API, and what it aims to do is to unify your existing messaging technologies. What do I mean by unify? It means not just to sit on top of them,
06:02
but also that you could use different messaging technologies in your infrastructure and in your deployment at the same time. So what we aim to do is to create an open source alternative to SQS and SNS.
06:24
Do you know these services by Amazon? Okay, so basically SQS and SNS are both services by Amazon. SQS is a simple queuing service, and SNS is a simple notification service.
06:44
The first one provides you queues, and the second one provides notifications. They are both separate services, and what we aim to do with Marconi is to supply you these two services in one single service, which is Marconi.
07:03
And we aim Marconi to be used by application running on the OpenStack cloud. So now I want to talk to you about a few really exciting use cases
07:22
we have for Marconi. So the first one is actually to deploy SQS using the Marconi service. So you will just have your own queuing service, and you will be able to sell queues to your users. Also, the rest of the use cases
07:44
are aiming to be used by OpenStack services. So the first one is the Horizon notifications. So when you start an instance, and whether it fails or succeeds, Horizon gets a notification.
08:02
So you are able to do it today, but we want, like, Horizon is just polling, but we want it to be able to get notifications. So it will be able to do that using Marconi. Also, we have the serial monitor service,
08:21
so it generates events and statistics based on notification it gets from the other OpenStack services. And currently, it does that using the centralized message broker of OpenStack. Sorry.
08:44
So it is using the MQP message broker, and it uses a really low-level API, and what we aim to do is to give it a more high-level API, so it sits on top of the messaging technology. And we want to allow guest agents intercommunications,
09:04
which means you have a guest agent running inside of an instance, and if you have a failure, you will want to be able to communicate to the other guest agents or to services in OpenStack. Currently, you are able to do that
09:22
using the messaging broker, but in this case, it's just not secure enough, so we want to introduce an alternative. So now I want us to move to a high-level overview of what Marconi actually looks like.
09:45
So Marconi's architecture is pretty simple. It's composed of three layers, the transport, the API, and the storage. The transport is the actual protocol
10:02
that the clients talk to, and the API introduces you to the Marconi resources and the actions you can perform on those through the protocol. And the storage are the actual messaging technologies that Marconi talks to,
10:20
the underlying messaging technologies we're using. So a really important thing I want to mention about Marconi's architecture is that it's composable.
10:44
So you can play with it, much like as if it were Lego bricks. It is obviously plugin-based in order to allow you to do this, and each plugin has to conform to a well-defined API. So you can just choose the transport
11:03
and the storage you want to use, and just use the suitable plugin. So on top of the transport layer, we have the authentication middleware. It is actually provided by a third party.
11:23
Currently, we support two authentication methods, which are Keystone and basic HTTP. So with Keystone, you basically have all the multi-tenancy features that it already allows. So you will be able to use multiple tenants
11:43
and multiple projects under the same Marconi deployment at the same time. Obviously, you can write your, if you want to, you can write your own authentication method. You can just write a plugin from Marconi to use it.
12:10
On the transport layer, we currently support, that's our production protocol, which is HTTP. This is what we use.
12:21
We plan to support, we target TCP for the June release. On the API layer, we expose Marconi's resources. So the messages are the main and most important resource in Marconi.
12:41
That is actually what Marconi is all about. It's about delivering messages. So these messages can be read, posted, and claimed from a queue, which is a logical entity. And you can claim the message from a queue. This means that when a worker claims
13:00
a message from a queue, the other workers can process those messages at that time. Also, you can configure all of your messages, queues, and claims in terms of TTL. So the storage layer are the actual messaging technologies
13:22
you're using, which you will deploy by yourself in your infrastructure. So currently, we support two messaging technologies. The first one is MongoDB, and the second one is SQL. We have both of these plugins already.
13:43
Excuse me. So SQL Kami is not really recommended for production environment. It's not really good for queuing systems because of its performance.
14:01
So what we use and what we recommend for production is the MongoDB plugin. It is currently used in the Rackspace cloud service. And we also target to have ready support for the OpenStack June release.
14:20
So MongoDB will allow you to have fully durable queues and really persistent queues. And on the other hand, you will have Redis, which will allow you in memory support for your queues. So if your application needs a really high throughput, it will allow you this.
14:41
So let's move on to how you can use and deploy Marconi in your own infrastructure. So we have two ways to do that.
15:01
The first one is using a single storage cluster. So you will have multiple Marconi nodes running in parallel in your infrastructure on top of a single storage cluster. This storage can be whatever you want, whether it's MongoDB or Redis
15:20
or whatever fits your application needs. You just have to write the plugin for Marconi. And you choose it according to your application needs. Another way to deploy Marconi is using storage pools. This means that just like before,
15:41
you will have multiple Marconi nodes. They're still running in parallel, but instead of sitting on top of only one storage cluster, they will sit on top of multiple independent clusters. So you can choose these clusters to be whatever you want.
16:00
And you can configure Marconi to talk to these clusters on a queue basis. This means that if you want your queue to be fully durable you will choose it to be on a MongoDB storage cluster. And if you want it to be in memory and to have a high throughput, you will use the Redis storage cluster.
16:33
So let's move on to the really great things about Marconi. So the first thing is that obviously it's open source.
16:40
So we all here love open source. It allows you to have a really simple unified API, which means you will be able to use multiple messaging technologies. It is also providing us with FIFO guaranteed for your queues
17:01
which is something that SQS does not provide. So obviously this feature depends on your underlying storage. Currently all of the storages we support are supporting FIFO guaranteed. And since Marconi is configured on a queue basis,
17:21
you can make sure that your queue will be FIFO guaranteed. We also provide you with storage pools, which allows you to use different messaging technologies at the same time and control the throughput of your applications.
17:43
Also a really important thing about Marconi is that it's really easy to scale. Just as long as you have enough nodes in your infrastructure, you will be able to have as many Marconi nodes as you want. And as we talked earlier about the use cases, we target Marconi to be used by OpenStack services.
18:04
And since it's really lightweight and easy to install and it plays nicely with everything else you have in your infrastructure, so it just fits in your stack.
18:30
So before we close, I want us to talk a bit about our plans for the future, our roadmap for the Juno release. So something we're really excited about
18:42
having are queue flavors, which are just like the Nova compute flavors that allow you to configure your instance. So the queue flavors will allow you to configure your queues so you will just be able to create a queue and choose a flavor, whether you want your queue
19:01
to be fully durable or to be in memory, you will be just able to choose a flavor and Marconi will choose the right storage cluster for you. Also, we aim to have live migration of queues, so in case you have a really heavy read
19:22
or heavy write queue and it's basically just killing your storage pool, you will want to be able to migrate it to another storage pool without any downtime to your application. Another thing we aim to do that we already discussed is having ready support that will allow us
19:41
in memory queues and for the future, we aim to have AMQP support. We have a lot of discussion about this topic upstream. It was also planned for the Juno release, but we had some problems, so if you have some knowledge around the area of messaging and AMQP
20:03
and unifying technologies, then we'd really love to hear from you and for you to join the discussion upstream. So after this talk, I really hope you got a clear view of what Marconi is and what it actually aims to do.
20:26
If you're already using Marconi, we'd really love to hear your feedback and if you have some new and interesting use cases after this talk, then we'd really love to hear your feedback. You can contact us.
20:42
So that's it. Do you have any questions? Any questions at all?
21:06
Okay. Okay, well thank you very much. Thank you. Oh, here we go. Oh, okay, yeah.
21:27
So currently, Marconi is using production in the Rackspace Cloud Service. We are not really familiar with any other production environments, but it is production ready, so you are just welcome to install it and try it.
21:45
Hi. So while I understand that Marconi is. Oh, okay, sorry. So while I understand that Marconi is supposed to be a core building block of OpenStack and to meet some special requirements,
22:01
what else makes it any kind different than other queuing solutions? Because you started your talk by saying it's not a competitor to, let's say, MPQ or Kafka or other messaging technologies, so how is it not yet another queuing and notification service in competition
22:20
with MPQ and Kafka and et cetera? So it is a queuing and notification service. It is not another messaging broker. That's what I mean by that. So it is basically just giving you another level of isolation on top of your messaging broker,
22:42
so your application will be able to just use a really simple and high-level API and not deal with the underlying messaging technologies API. So if you're currently using an MPQ broker and you decide it's not good enough for your application needs and you want to change it,
23:02
then you will just be able to deploy another messaging technologies like, let's say, for an instance, just install MongoDB if you want a really high and fully durable queue. And you will just deploy the new technology and you won't have to change any of your application's code.
23:22
Well, with MPQ being an open protocol, I can use RabbitMQ or whatever other service which uses whatever it wants as a storage with whatever reliability I need from that software. I know that Rabbit is the most popular one but not the only one. Yeah, so you will be able to use AMQP
23:41
and also other technologies. That's the purpose. Hi, sorry, when you say that Marconi could be used to let different open-stack services talk together, that means we'll have a new drive-in on slow messaging?
24:05
Actually, I don't really have a short answer for that, so I'd really like for us to talk about it later. Thank you very much. Any more questions?
24:21
Okay, well, thank you very much. Thank you.