Weave Net, an Open Source Container Network
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 490 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/46871 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Control flowExpert systemKernel (computing)Device driverSoftware developerWhiteboardScheduling (computing)Computer animation
00:37
Expert systemSlide ruleProjective planeMultiplication signSoftwareExpert systemComputer animation
01:33
Open sourceComputer networkInstallation artEnterprise architectureRevision controlSoftwareOpen sourceEnterprise architectureOpen setWindowRevision controlComputer animation
02:27
Computer networkModal logicGoodness of fitNamespaceVirtual machineSoftwareNumberQuicksortArithmetic meanPoint (geometry)Shape (magazine)Flow separationKernel (computing)DiagramBitComputer animationLecture/Conference
04:29
Computer networkData modelLine (geometry)Revision controlErlang distributionExpert systemEndliche ModelltheorieTheoryLevel (video gaming)Line (geometry)Revision controlCodeProgrammer (hardware)Row (database)Erlang distributionBitXMLProgram flowchartComputer animation
05:31
Revision controlVirtual machineEndliche ModelltheorieBridging (networking)ImplementationInjektivitätNamespaceWebsiteDifferent (Kate Ryan album)VirtualizationTap (transformer)Computer animationProgram flowchart
07:20
DemonImplementationAddress spaceMultilaterationTerm (mathematics)SpacetimeSoftwareComputer programmingDifferent (Kate Ryan album)Open sourcePhysicalismComputer animation
08:38
MassComputer programmingMemory managementSpacetimeKernel (computing)MeasurementSoftwarePerformance appraisalSet (mathematics)Computer animation
09:30
DemonDatenpfadModule (mathematics)SoftwareForceDemonRevision controlCalculationPlanningProcess (computing)CodeProjective planeBridging (networking)Open sourceKernel (computing)Similarity (geometry)Open setMeasurementGame controllerComputer animationProgram flowchart
11:22
DatenpfadPoint cloudService (economics)Exception handlingComputer animation
11:56
Scripting languageSoftwareRule of inferenceBitNamespaceGastropod shellTable (information)Bridging (networking)Computer animation
12:26
Continuum hypothesisMenu (computing)Complex (psychology)Scripting languageProjective planeScripting languagePhysical lawMultiplication signComputer networkOpen sourceAsynchronous Transfer ModeLine (geometry)Different (Kate Ryan album)Gastropod shellCodeComputer animation
13:55
EncryptionTable (information)Library (computing)Software frameworkRule of inferenceGoodness of fitQuicksortGreatest elementLink (knot theory)SoftwareOpen setFunctional (mathematics)Virtual machineBitData managementProjective planePosition operatorCryptographyCodeKernel (computing)SpacetimeFerry CorstenKey (cryptography)Game theoryState of matterPlanningFitness functionComputer animation
16:54
EncryptionTrailSinguläres IntegralCondition numberNetwork topologyLevel (video gaming)Internet forumSoftwareFerry CorstenCentralizer and normalizerPoint (geometry)Data managementConsistencyData storage deviceGame controllerPeer-to-peerInformationRootComputer animation
18:01
Network topologyData managementAddress spaceConsistencyIdentity managementSoftwareIP addressNumberData managementHash functionFitness functionRing (mathematics)Type theoryTable (information)Data structureCategory of beingSpacetimeCentralizer and normalizerLaptopNP-hardShared memoryInformation securityTerm (mathematics)Computer animationProgram flowchart
19:47
Default (computer science)Computer networkComputer networkAtomic nucleusNumberQuicksortRootEndliche ModelltheorieBitSoftwareMultiplication signProjective planeFigurate numberVirtual machineInterface (computing)Installation artFrequencyBridging (networking)Goodness of fitOpen sourceSet (mathematics)Rule of inferenceOperator (mathematics)Line (geometry)Default (computer science)StatisticsNetwork topologyElectronic mailing listTransmitterCountingProcess (computing)GoogolMereologyRoutingPrandtl numberData storage deviceComputer animation
24:57
Computer networkInterface (computing)Run time (program lifecycle phase)Computer networkRun time (program lifecycle phase)Projective planePlug-in (computing)SoftwareConfiguration spaceField (computer science)Interface (computing)BitProcess (computing)NamespaceSoftware maintenanceMultiplication signRevision controlDemosceneQueue (abstract data type)Computer animation
27:31
DemonDirectory serviceDirectory serviceComputer networkDemonProcess (computing)Server (computing)ConsistencyPlug-in (computing)Computer fileBackupCentralizer and normalizerSoftwareDifferent (Kate Ryan album)Set (mathematics)Multiplication signComputer clusterConnected spaceKnotComputer animation
30:19
Content (media)Point (geometry)Open sourceAddress spaceDesign by contractSoftwareImplementationPlanningGreatest elementProjective planeEnterprise architectureState observerCodeOverlay-NetzComputer networkRevision controlInstance (computer science)Line (geometry)RoutingComputer animation
33:16
EncryptionSinguläres IntegralCondition numberTrailCondition numberCASE <Informatik>Open sourceSoftwarePatch (Unix)Multiplication signRow (database)Link (knot theory)Address space2 (number)Direct numerical simulationNatural numberDesign by contractKernel (computing)Greatest elementStress (mechanics)Physical systemLibrary (computing)Computer animation
35:30
Computer networkComputer networkData modelCore dumpFacebookOpen sourcePoint cloudComputer animation
Transcript: English(auto-generated)
00:06
Well, thank you for coming. My name is Brian Borum. I work for a company called Weaveworks. So let me just check who's in the room. Who had heard of WeaveNet before they read the schedule?
00:21
About half the room, that's OK. And who identifies as like a kernel developer, or a device driver, or DPDK developer, or oh, they've all left, OK, that's great. So we can, because I put on this slide, I am not a networking expert.
00:42
I'm like a programmer, and I've been looking after this project for five years. It's been downloaded 250 million times, it's starred 5,000 times. There's certain things to be proud of.
01:05
But fundamentally, I don't think I did anything clever. So I'm kind of glad that all the clever people have left the room. I put my smiling face up somewhat because I'm going to put a bunch of people up.
01:23
This is a talk somewhat about the technology and somewhat about the people and the history of this project. So hopefully that's interesting. So what is WeaveNet? It's a container network, and I'll talk about that in a minute.
01:45
The primary thing we were aiming for is that it's easy to install, it just works. It runs anywhere, and there's a little asterisk because we mean anywhere that is Linux. But Windows runs Linux now.
02:01
We have tried this. You can actually run WeaveNet on WSL, so nearly everywhere. It's open source, it's Apache licensed. We never made an enterprise version. So enjoy.
02:23
Yeah, what is a container network? We had one definition from Justin Garrison, who works for Disney. I'm an admirer of his work, and generally the work of Disney.
02:41
It's good stuff. But he said there's no such thing as container networking. So that was a bummer, because I've been working on it for five years. But actually it turns out he uses WeaveNet, and it just works.
03:01
So yes. So more seriously, what is a container network? So the point of containers is, or one point at least, is isolation. Through namespaces and kernel features, one thing sort of believes it's completely separate from another thing, including network.
03:22
They have separate network namespaces. So now how do these things talk to each other? Well, whatever the answer to that question is, that's a container network. That's the definition I'm going to work with. Okay. Let's go back.
03:40
Okay. Let's go here. Conceptually, what does this mean? So I'm going to draw, I'm going to put up a large number of diagrams that look a little bit like this. So the meaning of the shapes, the big darker blue shape is a machine, whether it's bare metal or VM or something like that.
04:02
That's your kind of node in the network machine. And the light blue blobs are containers. Okay. So sometimes containers are talking to each other on the same machine. Sometimes they're talking to each other on different machines.
04:21
And by and large, we're going to have lots of these things, and they're all going to be talking at once in different amounts. Okay. So that's the kind of theoretical high-level model that we want to, that's what we want to do. Let's go back five years, five and a half years. This smiling face, Mr. Sackman, wrote the first version of what became WeaveNet.
04:52
He was out of RabbitMQ, in fact the founders of WeaveWorks all came from RabbitMQ. So here's an Erlang programmer, and the code, a lot of the code has quite a strong, it's
05:07
all written in Go, but it's a strong kind of Erlang flavor to it, which is kind of cool if you want to think about that. 3,400 lines was the first commit, spoiler, we're at like 30,000 lines now, so it's
05:27
grown a bit. Anyway, so basically what we do at that version, fundamentally what we do is we put a bridge, a Linux bridge on each machine, we connect all the containers on one machine to the
05:44
bridge, and then we tunnel the packets from one bridge to another bridge. So that's the conceptual model taken to an actual implementation. Let's take that down a layer further.
06:00
So specifically we set up a bridge, for each container we set up a Vith, virtual ethernet device, one end is inside the container namespace, one side is attached to the bridge in the host, and we listen on that bridge using PCAP. Whoa, the room went quiet.
06:24
No, seriously, we tested three different ways to do it, like tap devices and whatever, and as at five years ago in Go, as it stood, PCAP worked best. So there it is. So if you've got two containers on the same host, they're both attached to the same bridge,
06:43
they'll just talk to each other over the bridge, and WeaveNet doesn't get involved. If you have two containers on different machines, then we're going to pick up those packets, we're going to put them in another UDP packet, it's kind of homegrown encapsulation, and
07:03
we're going to send that over the network and deliver it to the bridge again via PCAP, packet injection on the other side, deliver it to the bridge, and that's going to end up at its destination.
07:22
This is, I like to call it at least, a distributed ethernet switch. WeaveNet implements an ethernet switch, it's a layer two network. It works purely in terms of MAC addresses, it does what a dumb ethernet switch does, it learns MAC addresses by seeing a packet come in, and it observes the source address
07:42
that it came from, and later on it has a packet to send to that destination, it uses what it learned, and it delivers the packets that way. So a physical ethernet switch is going to deliver on different cables, we're in software, we're
08:01
going to send to different hosts, but it's exactly the same concept. We also have the same fallback behavior, if we don't know where it's supposed to go, send it everywhere. That behavior will come in useful later. This is a real website, somebody was kind of, I didn't put their smiling face up.
08:26
Somebody was kind enough to make this website, pointing out something that we did actually know, but if I step back, can I step back? Yeah, okay.
08:40
So yeah, the reason it's kind of slow is, we start off in user space here, in the program that's trying to get some work done, we go down into the kernel here, we go back up into user space through PCAP, we put it in a UDP packet, we go down into the kernel again, across the physical network, and then do the same thing again. We go up, down, up, down, up, down, up, down, and yeah, it's kind of slow.
09:04
It's terribly slow. We used to measure it, like five years ago, we used to measure it at 300 microseconds per packet extra latency, and now you have to set this aside, what are you actually going to do with those packets? If the next thing that happens is they get delivered to a massive heap of PHP code,
09:23
then 300 microseconds is not your problem. But whatever, yeah, it's kind of slow. Okay, so next step in the evaluation, we implemented what we call the fast data path, because we have no imagination when thinking of names.
09:45
So kind of similar picture, the packet starts off in a container, again, we've attached a vth. The other end of the vth now is in a different device, which is an Open vSwitch data path. So this is implemented by a kernel module from the Open vSwitch project.
10:04
This is the only piece of the Open vSwitch project that we use. So we're in these kind of, these daemon processes are basically implementing our control plane independently of Open vSwitch, but we're using their kernel module.
10:21
So it takes the place of the bridge, at least in this version of the code. And we add a few kind of bridge-like behaviors to it to get everything we need out of it. But once a source destination MAC pair has been seen to be talking on the container network,
10:42
we set up a VXLAN tunnel, and that goes kernel to kernel. So the packets don't do this up, down, up, down, up, down thing. They are encapsulated, which costs you a little bit, but we used to measure this on a 10-gigabit network, which we thought was fast in 2015.
11:06
On a 10-gigabit network, we'd measure 8-gigabits of throughput. So it wasn't that bad. It's doing encapsulation, but it's kernel to kernel, and it's delivered to its destination pretty fast.
11:22
So the person that did this was Mr. Rag, Dr. Rag, I should say. Almost everybody that worked on WeaveNet has a PhD, except me. Sorry. Yeah, so like I say, I like to put up the smiling faces.
11:43
So that's the fast data path. Fixed our main kind of obstacle that we had in the marketplace, which was that it was kind of slow. Let's talk a bit about how we set all these things up.
12:03
So right from the very beginning, we need a bridge. We need vists. We need to step into network namespaces, step out again. We need to set up some IP tables rules. We need to set up some CISCables, a bunch of things we need to do. So how do we do all that?
12:21
In a shell script. We borrowed liberally from this project called Pipework by Jerome Petazzoni, who was at Docker at the time, I think.
12:42
And this project is a shell script. It turns out it's actually really concise to do the kind of IP, NetNS, blah, blah, blah stuff. So we had our own shell script, which is called Weave. And it started off of 350 lines.
13:02
And it has these commands like weave launch and weave attach and so on. At peak, it got up to 2,500 lines. It's not a very nice place to be to maintain a 2,500-line shell script.
13:22
So I sat down and reimplemented a lot of it into Go. So it's currently, I mean, this is the latest commit. It's at 1,600 lines or so. The features keep getting bigger on the option. It works in all kinds of different modes and so on. So that bloats it. But one thing is when recoding from these things
13:43
in shell script to Go, the code gets like 50 times bigger because Go is notoriously verbose. But there we are. So what else? We do encryption.
14:01
We do that both ways with the fast data path and the slow data path we rename sleeve because slow data path didn't seem like a good branding position corporately. You know, sleeve is a thing that encapsulates something. I don't know. Anyway.
14:22
So a cunning metaphor here. In user space, we use the NACL library. NACL, sodium chloride, salt. Yeah. Okay. To do our encryption, in kernel space, we use the XFRM framework
14:42
and there's a wonderful explanation on that link at the bottom. All the minute details of how we do this. One interesting tweak. We couldn't get this to work at all for months.
15:01
Essentially because the open vSwitch data path doesn't provide any way to drive the packets through the XFRM framework. We can't set a policy to say everything on this data path go through here. And eventually, the idea how to fix this,
15:24
we stole from Docker. We put all the packets through an IP tables rule which marks them and then set a policy on that mark. So we have an IP tables rule whose only function is to glue these two bits of software together inside the kernel that otherwise don't play.
15:43
And that's kind of... A lot of the history of this project has been sort of fighting with things that didn't quite want to do what we wanted them to do.
16:00
The history is there in the code and some of it I can remember. Anyway, so we encrypt the packets. We're doing key management up here. We did not roll our own crypto.
16:21
And yeah, some people like that feature. It's encrypted on this side. It's encrypted when it hits the underlying network. It's not encrypted here. So if you've managed to get onto the machine and you can sniff this vith then you'll see the plain traffic there.
16:42
But I always reckon if you've got that much access to be on the machine and sniffing a vith then you probably lost the game already. So who knows? What else? Oh yeah, Martinez wrote this. Martinez did all the gluing things together
17:02
at the XFRM level. He now works on Selium at iSurveillant which is that vendor. So that's Martinez. Okay, so change tack again. WeaveNet is a peer-to-peer network.
17:22
The title of this talk is No Central Point of Control. And it's a pun on the management style and the technology. We wanted it to just be install and run whether you're running it on your laptop or in the cloud or on a hundred hosts or whatever.
17:44
And what most people did to put together a container network was they rely on something like etcd to be a central consistent store of what's going on to hold all the container information all the routes, all the whatever.
18:02
And we didn't do that. So WeaveNet is completely peer-to-peer. You can start with one peer and you can start adding more peers. They talk to each other via gossip So I've given each one a little flag.
18:23
Each peer has an identity on the network. And that peer can be present on the network or it can go away you know, you can close your laptop and take it on a plane and open it up again and it'll still work on the network. So the way we do that is all the shared data structures are implemented
18:42
as CRDTs as eventual consistency data structures. They're specially designed so that we can do that like somebody can be absent for any number of hours and come back again and the data reconciles. It all fits together.
19:02
That is incredibly hard work. Anyway, it has this property that you don't have to set up etcd before you get started but it is very hard work. We do this for IP address management
19:20
as well. We do it for several things but one of them, so we basically take an IP address space and map it onto a ring like a distributed hash table type ring, that idea and then spread that across the network and gossip updates to that ring. That's how we do
19:40
IP address management. Yeah, okay I wanted to talk about the community a little bit. I think I have a chart here. When it says installs
20:00
we get a count from Docker of the Docker pool operations. It's running well over a million it was up at two million this is one year, the last year two million a week down to about one and a half million a week
20:22
We see this software fire up a lot. As an open source project we don't have a very good idea of who's using it. People write in when they have a problem sometimes but they generally don't tell us just that they're using it and they're happy with it.
20:43
So this is one of the few bits of evidence we have. The thing gets fired up in some sense a million or two times a week. Compared to that we get very few PRs
21:01
We get lots of people coming along and saying things like this is just one I picked on because it came up recently and this is over a period of a year and a half, people complaining about a setting and saying why don't you change the default it's one line send a PR
21:22
People don't know how, maybe So most of the work has been done by people being paid by Weaveworks This is the GitHub contributors list A fun fun statistic after being the lead
21:42
on this project for five years I'm the second highest contributor Matthias Radistock who is also xRabbitMQ he was co-founder of Weaveworks he's still the number one contributor but all these people work for Weaveworks Mike Bryant
22:00
is the biggest contributor who doesn't work for Weaveworks We do have a long tail of people who did manage to come up with one or two PRs which is great and I would like to encourage that but it is a little bit dispiriting
22:21
when people just want to complain about the software and demand that it does something else Kubernetes This is what you were promised This is the theme of this day
22:40
So Weavenet is quite popular with Kubernetes I thought I'd just kind of run through what does that mean exactly and what is it doing there and how does it work Kubernetes doesn't just talk about containers it talks about pods
23:01
a pod is a collection of containers on the same machine and so in Kubernetes world conceptually the blue blobs are pods but the same stuff is going on they're talking to each other and Kubernetes has a very small set of rules one of which is that any pod can talk to any other pod without going through NAT
23:24
and funnily enough the rules the sort of model, the networking model of Kubernetes matches very well to Google's network so I don't know if we'll ever figure out how that happened
23:40
but if you run at GKE Google's commercial Kubernetes then this thing with the bridges they just have roots IP root layer 3 roots from machine to machine so they have the same thing with the bridges but they don't have anything else other than the
24:01
Google network to transmit packets between machines they just use Linux routing and let the underlying network deliver the packets to a bridge at the other side and that just works if you Google it pretty much doesn't just work anywhere else
24:22
so there is a need for something to take that place and Weave.net is one of the things that people sometimes choose to take that place so back around the time this was getting popular which is about four years ago now
24:40
the project Rocket which came out of CoreOS which was kind of a competitor to Docker they had they had this very simple model for network interfaces where they would exec a process that would add a network interface so that became CNI
25:00
essentially some people including Weave workers got in a room and said yeah that should work and it got named and it got turned into a project and I am a maintainer of the CNI project
25:21
but the CNI is supposed to be really really thin I just thought I'd walk through what exactly that is so CNI is not coupled to Kubernetes it is like I said it came from Rocket it's completely independent of network and what we call a runtime
25:40
so Kubernetes is in the place of a thing we call the runtime and CNI speak and physically it's the bit of Kubernetes called the kubelet which is the bit that runs on on each node so the kubelet calls a CNI plugin and it calls it right now
26:00
the interface is exec it execs a process in the host namespace not in a container it supplies a JSON config which lists a few things out like maybe which subnet you're supposed to be using something like that the plugin then talks
26:21
conceptually you've got a network somebody showed up with a network you bought one from juniper or you installed weave net or you're using cilium somebody's got a network so the job of the plugin is just to be that little bit of glue in between to interpret this JSON spec
26:41
and to cause the network to attach itself to a container so that's that's the idea of the CNI project and I think it's worked fairly well in its goal of being agnostic and kind of
27:01
staying out of the way of people I do quite often hear complaints that CNI doesn't do this and CNI doesn't do that the unfortunate news is it's never going to do those things because it's trying to be the thinnest possible layer that could work for everybody this is JSON if you want to say extra things in JSON
27:22
just add them party on just add fields ok that's CNI how do we get weave net installed so I just mentioned the plugin runs in the host as a process on the host
27:41
and everything we're talking about is containers which are isolated so we get we get run that by devious trickery we mount a directory off the host and start when weave net starts up it copies the file into the host directory so now it's in the host
28:02
I as far as I know I invented this trick but everyone does this now so maybe I copied it from someone else tell me at the end if it was your idea yeah so Kubernetes has this concept of a demon set which basically means run the same thing
28:20
one on every node and restart it if it dies that kind of thing so that's how we fire up we just we arrange for someone to ask for that demon set that fires up a copy of our software on every node we do this trick with
28:41
copying a file onto the host and now we're away kubelet is now going to call the plugin plugins going to call backup into the demon and that's how that all works I observed
29:01
about not having any kind of central consistent idea of what's going on and of course in in Kubernetes you have exactly that you have the central what's called the API server does know everything that's going on in a Kubernetes cluster
29:20
and so a few times we thought about abandoning the eventually consistent stuff and just rely on what Kubernetes is telling us which is what everyone else does and never quite got around to doing that anyway it's an idea if you want to submit
29:40
a PR that'd be great we do we implement Kubernetes network policy which was mentioned if you were in a couple of the previous talks so like saying who's allowed to talk to who we do that by relying on what Kubernetes tells us because you know it's the only thing that knows all the labels on different things
30:02
and and somewhat excitingly the the network is implemented at layer two and the network policy is implemented at layer three and they essentially have no connection between them we just run them as two separate processes in the in the same pod anywho is that all I wanted to say?
30:20
skip over that yeah that's that's pretty much what I wanted to say does anyone have any questions yes I changed it from weeb at some point because ok so
30:40
well that's not a question that's an observation but the observation was that I oh ok do we have any plans to support IPv6 so weebnet has no support for IPv6 in two ways it doesn't support IPv6 inside the overlay
31:01
and it doesn't support IPv6 as a target on the underlying network so which of those two did you want? you wanted both of them and may may I ask a question I mean the whole point of overlay networks generally is that you have some problem that stops you just routing across from one container
31:20
to another and that problem is very often an addressing problem in IPv4 so do you know what problem in IPv6 you're solving by having a like why can't you just route between the two containers? ok
31:45
so you said your answer your point was that you need some pods now my suggestion is that all pods can have globally reachable IPv6 addresses so you don't need anything else you don't need
32:02
me to write any code because IPv6 will solve your problem alright I think we have to take that offline yeah I mean you know bottom line
32:21
nobody did why doesn't it support IPv6 because nobody did the work it's an open source project we've worked as a company found something much more exciting to do which is called GitOps and you should all buy that we never managed to
32:41
monetize we've net we never made an enterprise version we never we never found anyone that was for instance willing to pay us enough money to do an IPv6 implementation thank you for the question any more? one over here
33:06
question is what was the contract race condition? so I should put Martinez's smiling face up oh I'm pressing on the wrong button to put Martinez up
33:20
so in particular that link at the bottom is not the right one to look for I should have changed that sorry ok the the I'm just trying to see if I can give you a short explanation basically it shows up when doing DNS requests particularly on Kubernetes
33:41
particularly using the Muzzle C library and what happens is it does two requests at exactly the same time for the A record and the 4A record
34:00
notwithstanding the fact that we don't support IPv6 the the the two DNS requests go out from the same source address to the same destination address same source port, same destination port and they hit a race condition in contract and one of them gets dropped
34:25
no it's fixed in the Linux kernel yeah like I say we spent most of our time not on our own software we spent most of our time fighting other people's software including Linux and and in some cases fixing it
34:41
so that yeah so Martinez wrote two patches he found three race conditions he wrote two patches to fix two of the race conditions so this is a you can google like why do I see a mysterious five second delay in my Kubernetes system
35:02
so this is not the only reason why people see mysterious five second delays but it's certainly a very popular one the the nature of the the requests they are from the same source address, same destination address same source port, same destination port
35:20
and contract does not know how to deal with that well time's up, thank you