Okay, welcome. Thank you for joining. I'm Nico. I'm from Argentina. And I moved to the Netherlands in 2019. I'm living in Amsterdam right now. I've been a Python organizer for a decent amount of years. I'm an organizer of this conference. And I've been an organizer

of your Python for three years. Before that, I was living in Argentina, where I'm one of the founders of the Python Argentina NGO. I've also been named a fellow of the Pathiton Software Foundation in 2021. I also work. And I identify myself as a software

engineer, but I've been working really close to the meta and operations. So I've been a sysadmin, I've been all those kind of things. But I still like to say software engineer, because I believe that the proper way to do Infra is writing code. When I moved to the Netherlands, I joined Optiver, where I'm currently managing three teams, the Linux

networks and infrastructure platform teams, that our teams are working on things I'm going to mention a bit today. If you want to talk to me, please say hello, Pigmen Discord or Pigmen Person. I'm going to be here still until the sprint and Sunday.

I work in Optiver. Optiver is a proprietary trading company. We are market makers. That means that we provide liquidity to the markets, and we try to do that with the best prices that we can. We have around 1,600 employees around the globe. And we use Python a lot.

Those 1,600 are not all developers, right? For sure, we have traders, we have risk, we have our teams. And we use Python a lot for infrastructure, of course, for research, for trading. And as you can see, we have many offices. I work in the Amsterdam one. So what I'm expecting you to get from this talk. So first is, there is a lot of

applications of Infra's code that you can just go and read the documentation that are common. But in this case, it's like, we have a few limitations where we have to do our own premise of infrastructure. And that has a bit of a different challenge. It's also a bit, I think it's interesting to see how others are solving problems,

and see maybe you get one or two ideas that you can use, or maybe you get interested in the topic. And I think also important for me is like, when you think in Infra, you add the code next to it, right? Infra is not only people doing Linux shells, and SSH, and patching cables, that kind of things.

So if you go home with that idea, I'm successful. Sorry. So this is like a screenshot of the Optiver website. And the reason why I'm using it,

is that you see, 1986 was the first trade ever of Optiver. That was in the options exchange in Amsterdam. And in this picture you see, there is a trading floor, and there is a lot of people. And the way that trading was happening at that point was like, you know, screaming, I want to buy, I want to sell, this is my price, right?

And you saw maybe movies, or maybe real images of Wall Street, you know, big guys, suits, like pushing each other, screaming to see who will be first, right? Well, that changed the idea, right? Exchange looks a bit more like this now, right? There is still some people in the trading floor, but reality is like, all the other changes, all the things are happening more in Adatacenter.

And Optiver is a trading company that has been trading for since 1986, have to convert to adapt to move there, right? And how it works of it is, imagine that this biggest square is Adatacenter, where one exchange is. And how it works is like all the members are connected to the exchange system.

And imagine each of these small boxes, member one, member three, Optiver is one participant, right? And that's one small room where you have all your racks and all your things. You connect, the exchange is going to send you data, market data. And they do that in a way that everyone is receiving the data at the same time.

So it's equidistance, right? So no one has an advantage. So member three here, that is closer physically to the exchange system, doesn't have an advantage because the exchange is building in a way that everyone is receiving at the same time. And then you can send orders, they use multicast, that's our network protocol,

and then you can send back things over TCP and send your orders. So if I want to be a member of exchange, of course there is a lot of paperwork, then they give me my square, my space, and I go and I deploy my hardware and my things. Maybe a solution would be, yeah, and then I just connect a pipe to the cloud

and write some Terraform playbooks, and I provision all my systems, and that would be amazing, that would be really easy. But the reality is that you really don't want to do that. Why? Because you are doing trading and the speed is important. Depending on the exchange, we can be talking about milliseconds, right?

Meaning I receive some updates from the market, some market data, and I want to update my prices or I want to send orders to the market, right? That can be milliseconds, it can be microseconds, it can be nanoseconds, right? So any cable that will take you to the cloud only just going,

it doesn't mean if it's a round trip, it's already too slow, right? So unless someone can change the realities of physics and have a fiber that is much more faster, you have to build, ideally, your own inference side exchange, right? Maybe in the future the exchange will move to the cloud and then we can build in the cloud.

And now this thing, imagine this many, many times around the globe, right? So I don't know, 20, 40, 60 different physical collocations that you have to build and you have to manage and you have to operate. So to set a bit, I don't know, is anyone here in the room

working with physical infrastructure like a data center? Nice, a few, nice. So what you have to do, right? So first you get the space, right? And they give you space, it's empty, they give you power, they take care of your thermal things, right?

And this is already something important, it's not standard. Every single exchange will decide a different data center, there are different countries, different standards, the racks are 10 centimeters different, so it's already hard when you want to standardize who you do those things. Like they give you in one place, they give you this amount of power

and the other one they give you half. So if your standard is I want to have 20 servers in each rack, you can, you can only have 10 because there is no power, right? Then you need to put racks, you need your networking devices, you need servers, cables for power, for networking, for all kinds of things. You need your WAN connectivity because you want to connect all these places,

you want to be able to access the co-location to run your automation, you need to connect to exchange, and you need other devices, right? I don't know, this can be, terminal service is a good example, it's like an out-of-band device to operate your switches, that kind of things. And then of course you need firmware to be provisioned to your devices,

you need OS to be provisioned to devices, you need configuration, right? And that's kind of the base, I'm omitting the applications here. And then also I'm oversimplifying with this, so imagine, I mentioned nanoseconds, so you can imagine that there is some custom hardware,

right, there is some special devices, there is low latency switches, etc. But let's keep it a bit simple just for the sake of discussion. Before going on, how to solve, how not to solve? And I think the ones that were raised in the hand will reflect with me,

maybe one of you have experience on doing infrastructure in a bad way, right? So if we have to do 40 different co-locations and manage them, please don't do that in Excel, right? Don't put like an Excel, say one tab is my APs, the other tab is my host names, and this is called the cables are connected, right?

So or building some manually maintained diagrams, right? Like some visual diagrams that you have to keep updating manually every time, and you already know it's going to be updated, right? The second thing is be careful with the tribal knowledge, right? Or with conventions that are not easy to maintain, but they seem easy, right?

For example, encoding metadata in your cost name, right? These are all red flags. If you're doing this, something is wrong. If you have pet's names, well, maybe that's okay. I don't know if it's random. If you don't call metadata, imagine like before it's not easy, because you don't have like a nice system,

you say server number one, database zero, application X, right? Like you put all the information in the host name, and you say, no, this is great. Then I just read the host name and I understand everything. But then one day you change your system, and then your database is in the server next to it, and you have to rename all your servers, right?

Or the other one is like, yeah, server number one, use IP number one, server number two, use IP number two. So I ping the server number one, I know it's the top in the rack, right? And you say, oh, this is great, right? This is really easy. It's easy to memorize, but it's easy for you, right? And you are not the best source of what's your infrastructure, right?

And ideally, the tooling that you have should help you to understand those things. The other one that is a bit obvious, but I'm going to say this, don't do it manually, right? And don't go and build one by one, except it's small enough, you can do that, then great, right? I say, oh, I have five servers, who cares? I will write it in paper and do it, I don't know.

Maybe it's okay, I only change the servers once a year. But you do manually, that means that you don't have a standard, right? Because even if you have a standard and you wrote the standard, how you know the standard is really there, right? How you know that I didn't type server 001, and then the next one I type server 0002, right?

An idea typo and that kind of things. There is no way to have assurance, right? Or if you can do that, you need to do several iterations or go and check all the time, right? That's almost impossible. There is no good inventory, right? If you want to plug your monitoring system or you want to use Prometheus, you have to manually configure Prometheus, right?

So there is not really a nice way to have that if you're doing it manually. And then you're spreading the problem, right? Then your application layer, so someone writing applications in your infra, will still have to adapt to your Excel files to run their application, right? And that's, of course, a really bad idea

and you are propagating the problems to them. So let's not talk about a breaking framework, like one single server can have like five different frameworks and you will be like clicking and clicking one by one to do updates if you do it manually. Basically it's like a snowflakes everywhere and it's a bit of a nightmare for someone doing infrastructure.

Finally, there can be other requirements and I have one example that we have in Optiver. We are a financial company, there are financial regulators and one thing that we are required to have and we also want to have is a kill switch. That means like our system can go crazy, can be a bug, I can start spending millions per second, right?

So we want a way to say, yeah, I want to disconnect you from the exchange now. I kill this thing. How, and if you don't do that in a good way, you are a Wikipedia page tomorrow, right? Like you don't exist anymore as a company. So how much are you going to trash your Excel files, right? To get the list of things that you have to disconnect from the exchange.

Again, this is a bit probably not really required but I think it's good to mention what don't do. And this question I really like is, can you really from scratch, right? If you're afraid, when I asking this question, you're doing Infra, then you have a problem, right?

And Infra's code is a bit of a way to solve this. And honestly, probably no one can do this 100% exactly like, oh yeah, from zero to 100 but if you're in the cloud, probably you can do like 99, right? Like ideally. And if you're not in the cloud, you should be able to do this as best as you can.

Then what's Infra's code? And I told you about Infra's code, so let's explain what it is. It's a way to manage and do provisioning through code instead of manual process, right? That's one point. It's a machine readable definition

instead of interactive configuration tools, right? You have a way to consume that with software. You don't need a human to be clicking things, right? So you should be able to implement code to use your Infra's code data. I think this for me is a bit of a case.

You add reproducibility, sorry. You add a bit of speed because you can do it faster. Less errors, humans, we do errors by default, right? And that means risk reduction and in total it's a bit of a cost reduction. And the cost is not only money, it's also time, it's also head space, right? I don't want to be thinking on IP addresses in an Excel file.

I don't want to be thinking about IP addresses. I want IP addresses to be assigned by some code I wrote two years ago and not to be a problem. So I can focus on how I made my observability better, right, how I make my automation faster, how I do all the things. Infra's code will enable you to orchestration, right? So if you have a nice Infra's code solution

then that means that you can orchestrate. You can say, oh, I'm going to install all my collocations every weekend, right? I can operate my systems in a higher level, right? You have like a bit more of a power. Of course, it's standardization, right? Your environment is, you know, you are sure,

you have assurance that it's going to be in the way you want it to be. A bit of a machine to clone snowflakes, right? It's like you can have one snowflake slowly, right? With your hands and then you reach a point when you can just copy it, copy, copy, copy all the time. And finally, one comment. For me, it's as important as the whole definition

of Infra's code. It's why it's declarative versus imperative. So an imperative solution to the automation is just describe the steps to achieve a state, right? So say, okay, you write some bash script and you say command one, command two, command three, or maybe, I don't know, something like Ansible will run some steps and will create

some Jinja render templates. That has a problem. It's not good for humans that are trying to consume that. It's also not good for machines because to give you an idea of what's going to happen, you need to process that bash in your brain, right? You need to process that in your head

or you need to render those things in your head. And declarative is more, what's my intent desire, right? What's my intention? Who I want this to be? And then some code will make it reality. I think Kubernetes is a good example of that. You define your app and then Kubernetes know how to deploy the container. You don't say the steps for it.

This is a screenshot for Terraform website. This is a really common case. And I'm showing this because I think it's a good way to spread inference code, right? So you have Terraform that gives you a way to have like a high level definition. Then Terraform has the code. So there is code that implements the providers, right? The Terraform providers. And then you have this that is the Amazon

or Azure or any other cloud provider or provider that works with Terraform that they already have some kind of a standard for you, right? They already have like a solution. They exposed end points. They give you SDKs and they give you things that you can consume. And sometimes we don't realize how much

they abstract from us. And this, if you're going to be doing inference code on prem, you really, really need to know this because you have to solve a lot of problems that they are solving for you, right? And here you have, okay, this is a version of my code. I'm saying in which region.

And for example, teach you micro, right? So I'm saying, I want a VM that is micro. But that means that there is existing menu. You cannot get any random amount of course or any random amount of memory that you came up that morning, right? They are enforcing URA standard, right? They are abstracting for you a lot of problems.

And what they are taking from you is I'll say, okay, this is standard. You can have this kind of VM or you can have this memory or you can put servers in this way and you cannot do these other things, right? And I think it's good for a lot of cases, right? Because then they set you some boundaries and then you can move in between that. But it's always good to think, right?

They are solving for you all the physical realities. You're abstracted for it. You don't care about a switch or that kind of things. So let's talk now how we do that a bit with all of them, right? Without having the cloud. First, I wanted to talk a bit about the open source software stack. Because ideally, if I ask one of you,

okay, let's solve this problem. We have 40 other centers. I say, okay, let's see if there is already someone solving this problem, right? I think that it doesn't exist like a de facto. There is no Kubernetes of on-prem infra, right? And there are some solutions. Like I think Netbox and Nautobox.

That Nautobox is a fork on Netbox. Those solutions are interesting, but are solving only some part of the problems, not all the parts. And I think there are a bit of challenges on how much you can adapt. NAPL is a great resource from the open source community.

NAPL is a library to talk to networking devices. And when they do, that is amazing. They give you a common interface, right? And then you can contribute to them, but you don't need to maintain your own interface with all the different vendors. But for sure, it would be great to have one. I would hope, right? That would be really, really nice. Because what you have to do,

if you want to do infra scope on-prem, you have to implement your tiny cloud, right? And then you have open stack, metadata necessarily, raken. Take a look to all those ones, are interesting. We decided in Optivr to implement our own solution, right? Based on the situation that we were and the conclusions that we arrived to.

Before going to that, before writing any single line of code, you need to have a standard, right? And in this case, it's much more important because you don't have a standard. Because if you're in the cloud, the cloud is forcing a standard on you, even if you don't like it. Here, you don't have. So which color of cables are we going to use?

How thick are the cables, right? How many servers we put in a rack? What's our network architecture? How we're going to do routing? How we're going to do redundancy? How we're going to, what's our standard? We overprovision or we underprovision, right? So think on all those parts and think that your code, your infra code implementation

is the implementation of the standard. And ideally, you do that in a way that is not a 101, so you don't have to rewrite the whole thing every time you change your standard. There is a decent amount of coupling, but think on trying to have this with some flexibility. You cannot have only one standard because some physical realities are as low.

So now, let's take a look at our solution. And I will try to use the last five minutes to go through it. If you see this big squirt here,

think from there to your right. That's kind of Amazon for now. Don't think on what you're saying. So it's invisible for you, right? And in this side, you have a standard, right? A standard, say, how you think. And then you have a high-level definition or think on values, right? It's like input from my standard.

It's like, what's IP address space? What are the uplings? What are the serial numbers of the physical devices, right? And what's the standard version? Then when you have that, you have what Terraform is doing, right? What Terraform is doing is knows how to read that high-level definition and do all the API calls to the providers

in the order that they have to be done to create your infra, right? And we have that, right? We have our SDK. We have our command line. And then with that, we can interact with our infra. But then the hard part is in the other part, right? You need to implement your Amazon solution.

So what we did here is that we have our intent system. This is the reason why the name is intent is because it can be source of truth for a lot of people. But we intentionally say intent because this is our intention. This is how our infra should look. This is a Django instance with a process database. It's a pretty standard thing.

But what we do there is we model our infrastructure, right? And then you can call an API and say, I want to create a location. I want to create a rack. And I want to put a server, a switch. And I want to connect this server network card with this port. And then you do a layer two of networking and layer three of networking. And what you have is that when you run this part,

the code that... Oh, sorry, I have the mouse in the rows. So when you run this code that implements the high-level definition of the standard, it will do all the API calls and it will define whatever you need to build your location in this database. After that, we have our provisioning pipelines.

We use fast API and other Python things. And basically, this system knows how to read our intent and then push the configuration, the OS, the framework changes to the device, right? It also has something that I didn't put in the slide that, for example, you have the physical reality.

So that means that you need to run this thing for a first time. And then you export the data and you give it to your data center engineer team. Please go and build this, right? Crack the servers, connect the cables, and these are the instructions to do it. And when that's done, you can then say, okay, now install it. And then you have the assurance part, right?

So you can take a picture. We have a system that is doing collection of truth. It's taking a picture. And we have an auditing system that is comparing our intent with reality, right? And our intent is saying, this service will have IP number three, but it has one. It will create an event because there is a bug somewhere

or someone did something wrong. So as a pretty common example, something that we do is we will go to build a new data center collocation. Then we will work in this file. We will run our script. Now this can be Terraform in the future, right?

It can be, why not? Today we are still working a lot on this part, but probably we can just make it Terraform. But at the end, what's important is what's the concept? There is code that implements your standard and knows it has to create 40 servers in a rack and it has to connect them in some special way. And it has to do the routing in the switches, et cetera.

So we run that first thing. We export the data to our center team, they build. And then when we have some basic connectivity to the devices, we just call our provisioning pipelines and we push firmware updates and OS. And there is a lot of super interesting things that we can talk here.

Like, I don't know, like we are doing Docker images being flashed to bare metal servers. By the end, for me, what is important is to get this whole idea. You have like an intent that you need to maintain and you need to solve a lot of things that the cloud was solving for you before. I'm hurrying up because I want to get time for questions. So a bit of conclusion is, as I say,

there is not a factor. I don't know if that's possible. I hope it is. It will be really nice because each company is a bit different, right? But there is maybe for, there is some cases that are common enough that there is going to be a factor one. The Python ecosystem is great for these kinds of problems, right? Like if you're solving these kinds of things,

Python is the best, right? It's super easy, it's fast. You have these nice frameworks to build on top. It's a bit of like building your micro club. And for me, it's also really nice. Something that I really love about my job is I have a bit of the luxury that I have a problem that is big enough that I can write my own micro Amazon, right?

Because usually you don't have a problem big enough to do that. And that's a really interesting challenge, right? It's like someone did a lightning talk yesterday and was showing how to change the Python interpreter and this person was telling me later, you can write your own shell and that's something really nice to do because you learn a lot. And I think here, I learn a lot.

I have, for example, now a lot of thinking of how a cloud provider is doing things, right? And that was my last comment and I really wanted to give time for questions so I'm going to stop and ideally in the questions I can give more context. Thank you.

Thank you very much. Thank you very much, Nico. Do we have any questions? We have a microphone there in case someone wants to go and ask someone. Okay, we have the first person. Thanks. Go ahead. Could you share us about in any, like the major out that led to head

like what's the worst that ever happening when managing the data center either on the files or on the specific physical structure any horror story that you can share with us? Well, if you will be enough time in this industry, I saw fire, water, smoke, all the things.

And that's more because of physical realities, right? Like you're in the center and it's scary enough that you have to be ready to move to a new place, I think. I cannot think right now in one that I will be scared that we did but I can imagine our system having a bug and changing every single gateway

for every single server or something like that. Didn't happen so far. Like, please. Yeah, hi. Thank you first of all for that was really insightful. There's definitely interest in the space about the idea of these kind of on-premises in micro cloud. Recently, the HH has published

that base cam is no longer on the cloud. AWS had published that the serverless architecture doesn't work even for them. So my question will be, is there any plan on your part, Optiva, whatever, to open source this stuff or even kick off this open source movement

towards micro Amazon, as you call it? Thank you. You're welcome. I don't think right now we are planning to do that but for sure, so we try to contribute to the projects that we use like Napalm and some other projects. I think releasing the whole thing as a package is like you have to be the product that we are not ready to do because it's kind of our internal usage.

So it will be really hard. We will need like a whole team maintaining it to be generic enough for others. What I do expect that we'll be able to do is release some components. Like maybe there is a piece that is installing Linux in a server and it's a pipeline that is pretty easy to isolate. I think those are things that we can open source. And yeah, I hope we're ready to do that at some point.

Thanks for your talk. I have a question. I think that your system may be far from being transactional. My question is, how do you manage inconsistency? Well, I showed this auditing part, right? So if I understand what you're saying with inconsistency,

it would be like, my intent is different to my reality, right? Something like that. Like for example, if you order to create, I don't know, a cluster and first machine was created and the second one was not created. Yeah, so we run our pipelines in a way

that they are idempotent, right? So you can run the pipeline many times and it will try to achieve the end state. So if there is a bug in the middle, you will fix the bug and rerun it or you will find the problem and rerun it until you achieve the end expected state. And we use this feedback loop to check if that's the case.

Thank you very much. We don't have more time for questions, but there was one question that they already tagged you on Discord. So maybe you can follow up later on. So cool, let's thank Nicolas again. Thank you.