Thank you for coming. My name is Raphael Gomez. I work at Octobus. We're a small consulting company specialized on Mercurial. I'm going to talk about how we interfaced Python and Rust within Mercurial to...

Well, you'll see why. What are the challenges that we faced and how we fixed them, how we didn't yet for some. So, for those of you who don't know, Mercurial is a version control system in the same generation as Git.

It was made in the same month of April 2005. It's written mostly in Python. It has C extensions for performance reasons, mostly. It handles huge repositories for companies like Facebook and Mozilla with millions of files and revisions.

It has a very powerful extension system that is super interesting and I have no time to talk about it today. So maybe check that out for yourself. Why did we choose Rust? I just said that we had 40,000 lines of C code. Why move to Rust? Why use Rust in general?

As a recap, Rust is a low-level language with a very powerful type system. It has no garbage collector, which is quite important when you're trying to interface two languages. If both have garbage collectors, you run into some very tricky problems. So that's nice.

It has a lot of compile-time memory safety. Some people might argue that it's too much or too little. But it has more than C does. And it allows for simpler parallelism than if you were to write the same code in C or Python, for example. So compared to C, maintainability is a lot better.

It has a better signal-to-noise ratio. That means that more of the code that you write actually is the algorithm that you're trying to do and has less to do with just freeing up memory and getting segmentation faults. You have better compile-time guarantees, which boils down to the same thing.

You have a standardized and modern tooling in the form of cargo and auto-formatting and a test suite, for example. And it's safe by default, by some definition of safe, with a few escape hatches, which are very easy to grip, like the unsafe blocks and a few others.

There are many reasons to choose Rust, but one of the main reasons is for performance. It's comparable to C for sequential code, kind of. Parallel code is much simpler to write and to maintain. So for us, in a version control system, it's a very good asset.

It allows for optimizations and possible for C compilers. I'm not saying that Rust is faster than C, as it is. Sometimes it is, most of the time it's not. But it's pretty cool. So there was an experiment made by Valentine, who's a software developer at Jane Street.

And he did a very small subset of the agstatus command that was written in pure Rust on a big, big repository that they have. And as you can see, the performance is quite a lot better than what we have in the Python and C version.

It's orders of magnitude better. So that sparked a lot of interest in the mailing list and in the community in general. And the plan to put Rust inside Mercurial was already kind of in the works, but this kick-started a lot more work, work that I was a part of.

But just before I joined, Mercurial project chose to use Rust CPython, because there are many ways of bridging Python and Rust. And Rust CPython is the one that they chose for many reasons. I really don't want to get into them.

It compiles on Rust table, so there you go. It's composed of two packages, you could say, crates in the Rust ecosystem. It has a low-level crate for binding to the CPython ABI. So basically you would never interact with that. And a high-level crate to interact with Python, so to expose a module that looks like a Python module to Python,

create functions, classes, and execute Python from Rust, which means that you get an eval function. You could just put Python within Rust and just execute within a context.

So we have the following structure within Mercurial. I'm only talking about CPython now, so for people using PyPy, I'm sorry. So the pure Python code, of course, talks to its backend, CPython. The C extensions do the same thing. And Rust is split in two crates. The first one being hjcore, which is a standalone library that has no idea whatsoever that it's talking to Python.

And hgcpython, which is one of the crates that can glue hjcore to a Python executor. It's the main one that we're working on because CPython is so common.

And yeah, that's the structure. So I was very excited to start working on Rust at first, and it was not super fun for many reasons. The first non-trivial thing that I started writing was about twice as slow as the reference implementation, so that was not super cool.

And the reason why is because of friction. When you're trying to piece two languages together, no matter the languages, except if it's like C and Rust, you basically always have issues at two levels. You have the developer level, so with the complex interface code that you

have to write that is basically completely orthogonal to what you're trying to solve. You're trying to do a problem, and then you have to take your data structures and move it to and from the other languages, which you really don't want to have to think about, but you really do. And exchanging data in general is costly. Because you have, at the very least you're moving memory around, at worst you're allocating a lot of it, which is very expensive,

and then you have to loop over all your data. So, for example, if I am stating 100,000 files on my laptop with hot kernel caches in parallel in Rust,

it takes about 30 milliseconds, which is pretty cool. And if I give the results back to Python, it takes about 300 milliseconds for anything useful, so you just add up one order of magnitude more than on top of what you were actually trying to do, negating the entire point of doing it in Rust.

So we have a few possible solutions. You can just exchange less data. So you move up one abstraction layer and you, for example, instead of giving the bytes of a file, you give the file name and make Rust open the file, that kind of stuff, which is pretty much the same as doing more in Rust. And you can also communicate with C directly and not go through the Python layer.

So if you go back to the structure that we had, there's this arrow that we're trying to use, and it turns out that in the Python standard library we have something called capsules,

which are Python objects that encapsulate, that's the typo, function pointers. So their main purpose is to allow for a shared C API between extension modules. So that's what we're trying to do, because Rust targets the C API.

You can open your favorite Python interpreter and look into the data module and see a capsule object. There you go, that's in Python. But moving up abstraction layers, you need more powerful abstractions, and there were some missing features in our abstraction layer, which is rusty Python.

For example, there was no way of dealing with and creating sets. Python sets are a very useful collection to use. There was no support for capsules. So we kind of had to do that also.

There is then more hairy stuff. Inheritance for classes written in Rust. So what that means is if you write, if you create a Python class from Rust, so it has a Rust backend if you will, and you try to inherit from that in Python code, it will completely just crash at interpreter time,

telling you that it's not a valid base type. So that has to do with the fact that if you try to inherit from this class and you forget to call init, then what happens to the memory? Rust needs its memory to be initialized. It doesn't just work like that.

So you can't really extend a Rust-backed Python type. So you have to use composition over inheritance, which makes some people happy, but it makes the performance really unhappy, and it makes me unhappy because I have to do all of this manual encapsulation, which is slow and cumbersome.

Properties don't have properties yet in Rust C Python. They're very useful, and if you're trying to build a drop-in replacement of a class that you've written in Python and just trying to rewrite it in Rust and expose the same interface, you have to, again, do the same encapsulation mechanism

to have properties that call your inner methods in Rust and blah blah blah. Setadder is not in Rust C Python, which means that you cannot have instance attributes, which is very limiting. And the last one, but not least,

is when you're trying to build an iterator on Rust collections. So I'm going to talk about this one a little bit more, because it's more interesting and we've found a solution for it. The main idea is that you want to have a Rust collection, let's say a vector or anything that you can iterate on,

and you want to pass one value at a time when it's being iterated instead of just taking the whole collection, moving it through the FFR layer, having two copies of the same stuff and trying to synchronize it, and it's just there. So you want it to behave exactly as a Python iterator would,

but to do that you have to tell the Rust compiler that it really has to let go of the memory that it's trying to hold on to, and it basically revolves around sharing the references through the same object between the two languages. So, those are not my slides.

All right, that's okay. There's supposed to be a slide here talking about the work that we've done, but basically my colleague did a proof of concept in early June last year, and then I upstreamed the first, I would say, non-trivial data structure version of it

in Mercurial about a month later, and it's been upstreamed this week, yeah, this week in Rusty Python, so that's pretty cool. So upstream work, that's not up to date, because PySet has been done by my same colleague, Georges, and PyCapsule, same stuff, you have a useful macro to define capsules,

properties are being worked on, I believe, I don't remember who, but someone is working on them, and iterators and Rust collections are also a thing. So, this is our target. This is basically the lower bound of what we're trying to achieve,

because we know that this is unrealistic. It does not do everything that status does, and it's very focused on doing one thing exactly, and it's not very tested, et cetera, but it's a lot faster, and we know that we can do a lot better than what we're actually doing currently.

So where are we now? I have two cases, the first one being pathological cases in our favor, something that favors the newer code. As you can see, the new code using Rust is faster and has a good margin, which is nice, definitely,

but for a more realistic case, we see about 50% improvement in performance, which again, it's very nice, it's cool, it compounds nicely if you have a CI system or something, but it's very far from the thing that we're trying to achieve.

So, there's a lot more stuff that we can do to make this go faster. I talked about how Rust makes writing parallel code a lot better, a lot easier. We could do more things in parallel.

There are basically three main loops within status, and I think only one of them is run in parallel as of now, and it's really the simpler one, and it's not really the most expensive one. So just running those things in parallel, and say if you have 100,000 files, you don't want to check the first file, do all the things, then the second one, etc. It's very easy to parallelize most of the stuff.

Especially now that most computers have a lot of CPUs, it actually gets a lot faster. Better conditional execution. That has to do with the fact that Mercurial is a 15-year-old code base with a lot of features and a lot of bug reports

that have added little lines here and there and you don't want to break the backwards compatibility that you've had for 15 years with the huge test suite that we have. So you want to work incrementally when you're trying to put Rust inside of Python code, and one of the things that I've done,

because I have mostly been working on status, is doing the naive thing, just figuring out if it's any faster, if it's any faster, it's good enough, to work in a different iteration to get it to go faster again. So you can maybe think about the things

that are not useful to do in any given situation. Maybe we can optimize some paths and maybe some not. We can see. We can rethink the order of execution. That's something that Rust allows us to do because it has such a strong type system

that some invariants or some constraints of the problem we're trying to solve can be more easily fixable and reasoned about than if we were trying to use Python. For the very more complicated parallel stuff, maybe we could do two loops at the same time and do two things that weren't possible before

because it was just too much to think about doing. Of course, fewer exchanges between Python and Rust. That's pretty obvious. If you're moving fewer things and defining better interfaces between Python and Rust, maybe you have less overhead

for the foreign function interface that you're trying to define. And it has a better approach for maintainability in your project. It has a cost of defining strong interfaces. That way Python can do its job and does a very good job of using extensions and being the very productive language that we know,

and Rust can have its own job of being the very fast language that we know. Of course, the usual suspects of optimization. You basically allocate less in general, make fewer allocations even if you have to allocate the same amount of memory

using memory alignments, all that kind of stuff. And something maybe a bit more controversial in this room, not start Python at all. I'll get to the next slide. Python has a start-up time

that can take tens, even a hundred milliseconds sometimes. And if you remember the slide that I showed you, that's about the time that it takes to run status on a hundred thousand files. So maybe we can completely bypass Python in some cases

because Python has a very good place inside Mercurial and made Mercurial what it is now with its extendability and meta-programming capabilities. But if you're trying to run a CI, for example, and you're always asking for the status and the log

and the diff and whatever, this is not very, you don't really need customizability. You need something that is very fast and as fast as possible. And the start-up cost of Python actually costs a lot, just running Python at all. So maybe for some aspects we can completely counter it.

I don't have time in this talk to talk about that. Maybe we can discuss it later. Working so much with Rust has given me a renewed appreciation for Python. Code is very easy to understand. Even with a very weird Python that we have to write in Mercurial

because we're really at the edge of performance in Python, it's basically writing pseudocode most of the time and it's such an achievement in programming language theory to just have a language that most people can read without having to bother themselves

with a lot of details that they don't care about. You get something that works very quickly. There are some extensions in Mercurial. I said I wouldn't talk about them, but not just a little bit. There are some extensions in Mercurial that are about 100 lines, 150 lines that do a lot of work. It doesn't do it super fast at first,

but it does it at all and it's so great. So it allows for experimentation. You can figure out your interfaces later. It's very rare to break your entire project by just changing something somewhere in Python. Usually you can get away with it, whether that's a good thing or a bad thing is up to you,

but it allows for experimentation. And it is a lot faster than Rust code that you are not done writing because you have the 15th error message that tells you the same thing. So thank you. Do you have any questions? Do we have time for questions?

Do we have time for questions? Sorry. Oh, sorry. Hi. Thanks for the talk. I have two questions. One is related to starting CPython. I wondered is the code integration really worth it

or would it be better to just have REST-only comments and Python-only comments and just start one or the other depending on the Mercurial comments? If I understand correctly, the question is is the interfacing of the two languages worth it

or should we just bother with Rust or Python depending on the use case? I would be very happy if I didn't have to do any foreign function interface work, but it's the best case you have because having a complete rewrite of your software is usually not a good idea at all, so you want to work incrementally.

And that starts with just taking a small bit of your code and putting it in Rust. As I said, both languages have a lot of advantages and you don't want to give one up, I would say. I'd say it's worth it. We still have time for a couple of questions.

How do you test how do you test newer code doesn't break previous Oh, so how do we test that the new code behaves the same way?

You have to write new tests for Rust. Oh, yeah. How do I write tests for Rust? Yeah, and make sure it doesn't break anything. Most of our test suite is integration tests, so they talk to... Quiet people, please. They talk to the command line

and we see the side effects of what happens. So most of the test suite has no idea of what the implementation is. Whether that's a good thing or a bad thing is sometimes kind of tricky, but it allows us to change the implementation and just see if it broke anything. We also have Rust specific tests for smaller functions.

Most of the test suite has no idea of what's going on in Mercurial. Yep. You can just repeat the question. So, actually related to the first question, to avoid the FFI, would it be an option to make a server in Rust and then talk to that server

over a socket, for instance? So, the question was, would it make sense instead of FFI to use a server client with Rust and Python? Yes, and in terms of performance, it does not hold up. We're trying to shave off milliseconds

of runtime and going through an HTTP stack or a socket usually does not make sense. But maybe. I don't know. Maybe in some cases it could work. Okay. I think that's enough.