Using Rust for your network management tools!
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Alternative Title |
| |
| Title of Series | ||
| Number of Parts | 542 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/61643 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
SoftwareNetzwerkverwaltungComputer networkSoftware engineeringService (economics)Computer animation
00:36
Computer networkSocket-SchnittstelleNetzwerkverwaltungBinary fileLink (knot theory)Keyboard shortcutConfiguration spaceSpacetimeKernel (computing)Process (computing)Suite (music)Library (computing)Computer networkSoftware frameworkData structureImplementationTelecommunicationState of matterProjective planeComputer networkDefault (computer science)Data dictionaryFile formatComputer networkNetzwerkverwaltungComputer fileValidity (statistics)Data structureConfiguration spaceLibrary (computing)MultiplicationCondition numberSerial portFirewall (computing)Order (biology)Operations support systemSet (mathematics)Process (computing)Address spaceSpacetimeCoordinate systemAsynchronous Transfer ModeKernel (computing)BuildingIntegrated development environmentSoftware frameworkPhysical systemNumberLink (knot theory)Computer animation
06:47
TelecommunicationComputer networkLink (knot theory)InformationQuery languageFormal verificationReal numberInterface (computing)Configuration spaceStreaming mediaSocket-SchnittstelleLibrary (computing)Keyboard shortcutSoftware testingDisintegrationLibrary (computing)Configuration spaceConnectivity (graph theory)Streaming mediaInformationKeyboard shortcutFormal verificationComputer fileLink (knot theory)Reading (process)State of matterINTEGRALValidity (statistics)CodeSoftware testingNetzwerkverwaltungKernel (computing)1 (number)Multiplication signTelecommunicationCASE <Informatik>Projective planeDifferent (Kate Ryan album)MathematicsComputer networkPatch (Unix)Socket-SchnittstellePhysical systemReal-time operating systemSet (mathematics)Computer animation
12:59
Term (mathematics)Multiplication signMatrix (mathematics)Software developerExpected valueOperations support systemCompilerPoint (geometry)Software bugDampingState of matterWritingView (database)Keyboard shortcutINTEGRALComputer animation
18:31
Program flowchart
Transcript: English(auto-generated)
00:05
We have Fernando. Yes. Fernando's fine. It's been a long day. Fernando, he's going to talk about using Rust for your network management tools. Take it away. All right. Thank you.
00:23
All right, so welcome everyone. My name is Fernando. I'm a senior software engineer at DREHAT. I work for the networking services team, mainly focused on network management tools. And today we are going to talk what was our journey, building a Rust tool for network management.
00:41
So OK, we did not start with Rust. We started with Python. But after some time, we decided that we wanted to shift to Rust. So this is two talks in one. One is how we did build the project in Rust, and what we learned when moving from Python to Rust. OK, so network management.
01:01
What's network management? Basically, it's all the operations that you do to configure your networking. Roots, interfaces, DNS, firewalling. Whatever you do, it's network management. So it's a process that is quite complex, because it requires a lot of coordination between user space and kernel space. We need to check when we get notification for kernel space,
01:22
because all the tools could modify the network status. We also need to communicate with kernel in order to configure stuff. So it's a quite complex task. And there is already a tool, which is network manager. It's by default the tool that is in almost all situations
01:43
used for managing your Linux network configuration. And we were willing to use it, and we were willing to build on top of network manager, because implementing everything was really, really complex. So we created NMS state. And NMS state is a tool that communicates with network
02:01
manager, and it's a library with a command line tool, and allow us to configure the network using the creative states. So you can define what do you want, and you don't need to care about how is network manager or how is the kernel doing, and what it's going to do,
02:22
or what are the dependencies. You don't need to care about any of that. NMS state is going to manage it, so it makes everything easier. So as I say, we started to build NMS state in Python. And one day, we noticed that a lot of our users were willing to chip a binary and not Python,
02:41
don't use the Python environment. And well, there were also some performance issues, because we needed to do a lot of operations. So we decided to give it a try to Rust. And we have a problem, is that we have a library and a binary, and we needed to move both of them to Rust. And also, we already have a big base of users.
03:04
So we could not break them, and we need to do it in a way that we needed to support all the features that we already did in Python. So well, we created our own NMS state library in Rust, I will tell you how, and also the NMS state CTL tool,
03:22
which is the command line tool. All right, so the first thing is that we are using YAML files and JSON files, and we are parsing them. So in Python, this was quite trivial with an schema, and we needed to find a way to do it. In Python, we were using dictionaries so the user could create a dictionary.
03:41
And using a YAML library, it was quite trivial to convert that YAML into a dictionary. And we needed something in Rust to do this. So we end up looking at CD. CD is a framework for serializing and deserializing Rust attack structures efficiently and generically. We use it for YAML and JSON, but it supports other formats.
04:05
This allows us to keep our declarative state, keep our API, so that was pretty good. And we noticed that CD allows us to implement our own serializers and deserializers. So that was also a big plus, because we
04:21
could do validation steps and simplify the validation work when getting the configuration file from the user. And then there were a lot of decorators on server, so it was quite good for creating aliases, for creating multiple helper functions,
04:42
and also some conditional deserialization and serializations. So here is an example. For example, this is an interface state for a bond, kernel bond. And we basically defined it is up. It has an IPv4 address with this address,
05:02
with this prefix length, and it is enabled. And then we defined the link aggregation options. So we have the mode options and the ports. One really good thing that we have is that we have partial editing. So you can define what you want to change, and we are going to merge it with what you already
05:23
have configured on the system. And of all the decorators that you can see there, we were able to use the decorator for, for example, accepting numbers as a string, accepting strings, accepting only the number, custom strings, creating alias, renaming, all of that.
05:43
And it was quite good. So OK, we communicate with network manager, and we communicate with network manager to configure the network state. And we have a problem is that before we were using the libnn bindings, Python bindings,
06:02
and they were not available in trust. And we tried to create trust bindings, but it was quite complex because they used gobject and we did not have gobject, and it was a big mess. But we noticed that network manager is providing a DBAS API. So we said, OK, let's use DBAS then.
06:22
And we noticed that there is a create, which is ZetaBus. And with ZetaBus, we were able to communicate with network manager using the DBAS API. And with the ZetaBand, we were able to encode the data structures that we were using to communicate with network manager
06:42
and configure the settings that we wanted. So using this, we solved one of the problem, which is telling network manager what we want to do and also fetching what already network manager have, which is also important because, all right, there are some options that maybe we do not
07:00
want to override because the user configured it that way. And for patch LED team, that is important. We need to know what the user configure and what the user wants to modify. So OK, one problem solved. Then we have another problem. So network manager does not provide at all real-time information from kernel. And we needed that because we also do verification.
07:23
So when you configure something, you, in a mistake, do a verification step, which what it does is compare what the user defined, what is configured on the system. And we have a problem because network manager was not providing real-time information. And sometimes, it took quite some time
07:44
to get the information that we wanted. And we were having some problems on the verification. So we were looking for a library. And we did not find any library that satisfied our requirements. But we noticed that there is already a rasnet link library.
08:02
And the link is a kernel API for communication between user place and kernel. And also, I think, between kernel components. And it was perfect. We could use rasnet link, which is an existing library, to build another tool, which is NISPOR. So NISPOR only queries information from kernel
08:20
and shows you in a YAML file or basically proper data structures. Well, it was quite good because we started to contribute to rasnet link because rasnet link was an independent project. And they didn't support everything. So we were able to help there.
08:42
And currently, a lot of people use rasnet link. And it's a quite big project and probably the one that most of the people use when need to work with net link and rest. So we have one more problem. OK. Now we have network manager working.
09:01
We have verification working, validation working. We can read the configuration. We can do a lot of stuff. But then networking is complex. And there is one thing that is called OBSDB. And network manager configure OBS. But they do not configure global OBSDB settings.
09:22
And that was a problem because we wanted to do that. So how we did? We basically started to use sockets and using the rasstd library for stream sockets, we were able to communicate with OBSDB, send petitions,
09:44
read what they already have stored on the OBSDB, and configure whatever OBSDB settings the user wants to configure. So we created our own cell to JSON. Using cell to JSON libraries, we created our own JSON RPC to communicate with OBSDB.
10:02
This is internal of enemy state. We have considered to put it on a separate crate. But we did not yet. Then we had another problem. OK. I promise this will end. We are going to have a solution.
10:22
It will stop at some point. So we had users. The users were using our Python library. And some of them were willing to move to Rust. Some of them were willing to move to Golang. But we were already developing a Rust solution. And some of them didn't want to move from their Python code
10:42
to Rust. So what we did is create bindings. And we create plenty of them. First of all, we created C bindings. So C users could use the Rust library. Then from the C bindings, we created the Python and Golang bindings.
11:03
Finally, one of the other problems that we had is that we got a huge integration test base. And we wanted to reuse them. So with the Python bindings, we were able to integrate the Python integration test that we had into our Rust library.
11:21
So it was quite cool. Because we were able to start building the new Rust NMS state, but at the same time using the Python integration test. And this way, we were sure that we were not breaking any existing use case that we already support.
11:41
So that's it. It was a success. We are quite proud, because most of the people that were using it liked the idea. And even the ones that did not care about if you use Python or Rust were happy, because the change was completely transparent for the final user.
12:01
If you were using Python, nothing will change for you. The code is the same. You don't need to do anything different. So it will be a transparent update. And if you are using Python and are willing to use Rust, OK, you need to change your code. But basically, the API is the same.
12:22
So well, you were able to use the same YAML files and the same JSON files, and everything will work. So we got a lot of adoptions. And right now, the user base of NMS state is still growing. And we are quite happy.
12:40
Also, we created Golang bindings, because OpenShift people and Kubernetes people were willing to use it, and it's written in Golang. So we provided them with Golang bindings, and they really liked it. So yeah, it was a success story. So basically, that was our journey.
13:02
I would like to hear. I think we have time for questions. So please, ask whatever you want. I promise you that there are no dumb questions. So thank you very much.
13:23
All right, any questions? OK. I wondered what your experience was in terms of time to implement in Rust versus Python from a developer point of view. I think it took us around two years, two developers mainly
13:44
working on it. It was full time. It was a long journey, but it helped us a lot, having the Python integration test, working with a new library, because we were sure that we were not breaking the existing cases
14:01
and speed up the things a little bit. Do you have a feeling for how long it would have taken you if you had re-implemented it in Python? I mean, I know that's not really a thing, but roughly how long if you had said right? Going back to Python. No, if you had said, right, we've got it in Python, but for no good reason. We're going to rewrite it from scratch in Python to make it cleaner, let's say.
14:21
Just as how long does it take to write something in Python versus Rust, or maybe it's not possible to guess? Well, I think it depends. In my opinion, and this is a personal opinion, writing Python is much easier, but then you have more bugs. At least this was my experience. When I implement something in Python,
14:42
I do it in, I don't know, 30 minutes, one hour, two hours, but then I go back. And when I'm doing Rust, it took me more longer, a lot of compiler errors, a lot of unsafe stuff everywhere, so we need to avoid that. But then it's quite stable.
15:01
And I can say that nowadays the Rust version, even it is younger that the Python one is more stable. We got less bug reports, and we have more users. Thank you.
15:30
Did you run into any problems in terms of compatibility when you created C bindings from the Rust code? No, not at all, to be honest.
15:41
We did not have any problem. It was quite straightforward. We did not have any problem. I must say that the NMS library, where we spoke to the users, is quite simple. So that makes it simple for us. We did not have any problem. That's it.
16:01
OK, thanks. Before. You mentioned that it was a long journey when you
16:21
were implementing this in Rust. And could you compare what you have expected in the beginning of this journey and with the reality? OK, I must say that I'm not the only one person working on this. That is my teammate, Chris. And Chris was the lead here. And I must say that I did not trust very much that we
16:42
were able to do it in two years. So we were like, yeah, in two years we are going to have Rust. And I was like, I don't think so. But he was right. So I think my expectation is that it would take much longer. But it was much simpler than what I thought.
17:00
So also, I thought that we could have more problems with finding the libraries that we need to do all the operations that we needed. But I must say that Rust have a great ecosystem. And the libraries that we are using, they are really, really well maintained. And that's great. This worked for us. We have a question from The Matrix.
17:23
Sounds a bit weird. Tanya's asking, how long did it take your team to learn Rust? Or did they know Rust already? No. We did not know Rust. I mean, were. We didn't know what Rust was. And we did some work on Rust. But we did one thing here. We started with NISPOR instead with NMS state.
17:42
So when we noticed what are the missing pieces, we first started with NISPOR, which is much simpler than NMS state. And we learned on the way. I must say that I am also surprised with all the Rust resources, that it was quite easy to learn. But we learned on the way.
18:00
When we needed something, we started learning it. And then we revisited the code, and we changed things. For example, initially, we did not understood correctly how to use traits. So we did not use them. And then we noticed, right, traits are really useful. Why are not using them? And then we started to implement traits everywhere and make it more flexible.
18:20
Thank you. There's no more questions. Thank you for your talk. Thank you for listening.