I work for Procore. I've been a web developer for about 20 years, and I've been a Rails developer since 2008. And this picture was taken by a colleague of mine and is from our office. I kid you not. I love our office. I take a walk out here every single afternoon.

But for some of you who caught John's talk yesterday, you know a little bit about the history of Procore. But what we are is cloud-based construction management software. We, as our tagline says that I really love, we build the software that builds the world. And I'm up here today to tell you guys about an endeavor

that Procore did in 2016 and on into this year, which is Procore decided to really double down and build out a complete API across all of our projects and our applications and our tools. To give you a sense of what that endeavor entailed,

I want to talk a little bit about the guts of Procore for a second. Procore has, we're growing like crazy. Last count, I think we have over 700 employees, about 120 engineers, probably more than that, and at least 22 squads. These are probably old figures.

Our Rails app is over 10 years old. We have over 40 distinct tools, and each tool is basically an application unto itself. And we have over 500 controllers in our application. Procore is big. Our application is big. It's all a Rails app. It's huge. Forget majestic monolith.

We are a majestic uber-lith. I actually thought this would be a great name for a metal band. But beyond that, we operate with squads, and something that Procore really believes in is autonomy, not just the squad level but at the personal level.

We really believe that we should give direction to people and let them use their experience and their knowledge to get us where we need to go. So we strongly believe in that idea of autonomy, which is something that we tell to every new hire. It's a strong part of our culture, and it's something that we believe very strongly in.

So let me talk a little bit about that autonomy. We follow the Spotify squad model. Essentially, we break up our R&D department into squads that each have a product manager, a UX designer, a QA, and a handful of engineers. And these squads have ownership of certain tools

that they have basically full control over. They have a product ownership over. They choose the direction of that product. They choose what to work on next. It's that autonomy idea. And if you guys aren't familiar with the squad model, squads roll up into tribes. And then for things that go across squads for common interests,

we roll up into guilds. An example of a few guilds that we have at Procore is we have a front-end guild that takes on a lot of the front-end concerns of how we make decisions there, a performance guild that keeps an eye on our performance and how we're doing on our controllers and our database. We even have a guild for master failures.

With an application as large as we have, we have a lot of errant failures just from machine hiccups. So we have this model that has at least 22 squads, right, each owning their own tools on a big application. And we have to make a consistent effort to make one large API across the entire application.

How do we do that without it feeling like we're herding cats? Well, it wasn't always a smooth and perfect ride, but we got there. And I'll show you a spoiler. This is the end result. We've got a full, consistent, beautiful API. I think it's beautiful, you know, but kind of like I'm a mom here.

But we essentially accomplished what we set out to do. But before I get into how we did that, I want to talk a little bit about why we did that. Why did Procore double down and invest in an API? Well, there's a lot of benefits to building out an API like this,

both internal, which relates to your developer happiness or your code health, and also external, which is relating to your customer happiness and your sales health. One internal benefit that you can get is building an API forces a natural separation of concerns.

You now have essentially your API layer separate from your view layer, and that forces modular architecture, and that modular architecture is innately cleaner architecture. Another internal benefit that we get is change tolerance. We get, as DHH I think says,

a new Rails framework every three months. But if you build a good, consistent, well-designed API, it can last through several iterations of different JavaScript frameworks or front ends. You can replace that at any time, or you can integrate with any system, and your API remains relatively the same. I like the analogy of TV.

We've had televisions in our living rooms for almost a century now, and the API for how a television gets its power, essentially the power plug, has remained the same even though our televisions have drastically changed shape. So when you design it well, it can last for a while. One big external benefit

is that APIs engender customer trust. If a customer is looking at different products and looks at your product and has a well-maintained API, it makes them comfortable knowing they can get their data in and out of your system easily. That's going to really help close the deal, and it's going to really help engender that trust

between you and the customer before they've even engaged with you. Another is that an API can greatly expand your capabilities. When you empower other people, integrators, developers, or just people who are interested in your business space to build on your platform, you can greatly expand what your application can do

without spending any extra resources. And when that works and that works well, and I think it works well at Procore, you get to become the ecosystem. You now have fostered an entire ecosystem of applications that work on your platform, and everything just works nice, and you get to enjoy and reap the benefits of that.

So talking about the benefits, let's talk a little bit about, well, what makes a good API? When we're starting this endeavor, what's kind of the end goal that we want to look for? Well, a good API is predictable and consistent.

Developers don't want to have to write new code for every endpoint you surface. And especially, this compounds the fact, when you think about developing SDKs for your application, the more inconsistent your endpoints are, the more code and the more cruft is going to be introduced trying to take on those endeavors. A good API is static.

If your API changes in a breaking way, congratulations. You've now just broken everything that is integrating with it. One joke that I really like is writing an API is like sex. Make one mistake and you're stuck supporting it the rest of your life.

A good API is also simple and clear. Writing an API is not the time to be clever or show how good you are at writing code. An API should give back exactly what somebody expects it to, nothing more and nothing less. A good API is also flexible.

And this may seem right at first glance that it conflicts with the second point of that, how can it be both flexible and static? But think about steel, right? The way you make steel not break is make it flexible. And that's the same thing with an API. So we talked about what a good API should look like. Now let's talk about how we actually did it.

How did we build this big, beautiful beast? Well, it wasn't easy. It wasn't always easy and it wasn't always pretty. There was a lot of discussion. There was a lot of arguments. We have a lot of really smart and talented engineers at Procore and some of them have different ways and different views on how to do things. So there was a lot of arguments. There was a lot of style discussions.

There was a lot of architecture discussions. There was a lot of conflicting ideas and a lot of like really passionate arguments. But one thing that it boils down to, and it's something that Jeff Bezos says that I really buy into, is disagree and commit. You're always going to have those kind of things that it seems like an impasse, but somebody's just got to give

and you've just got to, for the sake of productivity, decide, okay, we're going to go with this. I don't think it's the right way. I believe it's this way, but I'm going to commit to this decision and be behind us 100% on this. So I want to step back a second and tell you a little bit about what we did to the squad model at Procore.

A few things, new endeavors that we're doing that really help this endeavor. One of the things we've added is the idea of what I like to call guide squads. A guide squad is essentially a squad that doesn't necessarily own its own tools, but really owns the process or things that stretch across other squads. It's like a guild, but with more ownership.

For example, I am on the API squad at Procore, and whereas I don't write a lot of the endpoints, those are owned by the developers who own the specific tools, we act as kind of a shepherd to the API development process, helping to make people disagree and commit

and to come up with those decisions that are really important to keeping up our productivity. So what does that mean? Like what do we do as a squad and how do we help this endeavor? Well, one, as Steve Krug famously says, don't make your developers think on things that don't matter to them. That's a big one.

One of the biggest questions that I get at Procore is, you know, should I do this or this? And it doesn't matter to them. They just want to know how to be consistent with the rest of the application. So we want to make sure that they're not having to think about these problems for things that they don't care about so they can get to the problems that they do care about,

like their tools and their endpoints. So one of the ways that we solve this is have a style guide. It goes back to the Rails concept of convention over configuration. We at Procore take this and we've built a style guide for our API, and we have style guides for a lot of our different sections

of our application. Now, I want to make sure to know that you don't have to have a style guide planned out all in advance. As a matter of fact, I think that's probably a very bad idea because you don't know what those decisions are all at once. But what I encourage you to do is keep a wiki or some kind of way that every time a question comes up

and you answer it, you write it down. It's very important that you write it down because you can't come to a decision and then walk away and the next day, how many times, it's like, wait, what did we decide there? Like, write it down and then after a while, guess what, you've got a style guide. Ours looks kind of like this. This is just a small section of it.

We like to keep things as pretty as we can, so it's kind of formatted. But this really, really helps, especially for new developers who get on our application and start writing APIs to have a lot of questions answered before they have to go and talk to another developer. Another key thing is boilerplate and examples

because, let's be honest, when we're writing code, we're copying a lot already anyways, right? So might as well make sure that we get the source as close to what we want the end to look like as possible if we know what that should be.

So here's a scenario that you guys may run into. We did a lot. Let's say you've got a developer writing an API. So he starts hammering away writing that API and then you've got your front-end developer who's there just waiting on that API or this could be your mobile team, your integrators, whatever. And finally, OK, he's done with the API.

He ships it over. All right, we've got it. We can start working. But then he realizes, oh, no, it doesn't have half the things I really need. And so then the back-end developer goes back and he's hammering on the API. He's adding all those scenarios and he finally ships it to the front-end developer and then the front-end developer celebrates and then the product manager goes, oh, wait,

we have customers who really need it to do this, this, and this. And then you're back right to the beginning again. So how do you subvert that process? How do you get rid of that? Because you don't want developers ever waiting, right? You want to be productive. Well, one of the big things that we do is we make contracts, API contracts.

And for those of you who don't know, an API contract is just essentially an agreement between your stakeholders about what that API is going to look like when it's finished. In our case, we just use JSON files. An example would be something like this. This would be for a simple to-do list application,

which we love in the Rails community, right? And we just, we pass these around and we get agreement and buy-in and the front-end developers, your mobile developers, your integrators can look at this and go, okay, this has all the attributes I need, or I need this, this, and this. You can cut through that conversation a lot faster

and not only that, but when you've made that decision and you've all committed to the central contract, then your front-end developers can start developing against it while your back-end developers start writing tests against it. And they develop and they come to the middle and they know exactly what the middle looks like. And it works and it empowers that parallel development.

So I want to go back to talk a little bit about building static APIs flexibly because we found that there were a lot of times where we were building endpoints that didn't actually meet the customer's needs. And sometimes you can't get contracts out to that, right? Sometimes it's not feasible to figure out all the things that your endpoint needs to do

ahead of time. So one of the things that we do at Procore is that we introduce the idea of support levels for our APIs. Essentially, we call them alpha, beta, and production. And a lot of times, if a developer is writing an API for a tool but he's not sure that it meets all the needs of the stakeholders and he's not able to write a contract for it, he will mark that as beta.

And the internal customers or the external customers who want to jump on these beta programs can come in with the full knowledge that this could change and these are kind of up in the air and then they provide that feedback, right? That we really want from a product perspective. And what that does is that takes that process which is instead of just like putting up steel

but more like pouring concrete. We have a little time to massage it after it's poured before it sets. And that's essentially what we do. And once we have all of that feedback then we promote it to production and we've got our new endpoint. Another important point

and I'm going to dive into this is use reusable components. It is impossible, in my opinion to build a consistent application without building some common components. So let's go back to our to-do list app. This is just a simple index action for a single to-do list

to return the items. But let's say we now have a product requirement that we need to be able to filter this endpoint to whether the items are completed or not. Well, it may look something like this. That's a lot of code.

We're stepping through this. It's not the prettiest, right? But this could happen. But now think about this. This is just one filter on one endpoint and we have over 500 controllers on our application. We have a lot of tools. Very big. Suddenly your application is now just full of cruft

and it's everywhere. So how did we solve that? Well, we built a gem that we call filterable. And we include that on our controllers and it's a really simple, really nice interface. You just include the concern and then you can say filter on your attribute and you can give it a type to validate again.

So now you have filter validations for free as well. This means this is much cleaner code, much more readable, and it moves all the complicated bits to a central location that you can provide full testing around. And not only that, but most importantly, all of your endpoints across your entire application

now filter exactly the same way. So you've gained all of that consistency for very little cost. We've also expanded it to support sorting and it also provides data types as well as scopes if you need more complex filters.

And I'm happy to say that we're going to be releasing this gem as open source to the Rails community. So another very important part of developing APIs is serialization.

In my experience, that's been one of the biggest performance costs for APIs. It's a constant problem and that's where a lot of people new to API development really struggle with. We were using JBuilder for a while, but JBuilder is very flexible, but it didn't have the performance requirements

that we needed and we found that it didn't really serve our needs in terms of organization. So we went with ActiveModel Serializer, which we found to be, for our instance, a better organized, a bit more extensible and faster. For those of you who aren't aware, ActiveModel Serializers is a gem supported natively in the Rails application.

You just throw it in your gem file and you can implement a serializer just like this. You explicitly list out your attributes and it also supports relationships. And if those relationships have serializers, it supports those as well right out of the box. And the beauty of this is because it's supported in Rails,

you don't even have to change your controller code. If it sees a serializer with the same name as your model, it'll automatically use that and you get the benefits of it right away. A sample output for that serializer you just saw would be this. You can see the attributes there. You can see that it included the association.

It's just, it's nice. It works. But let's add another endpoint to our API. In this case, we're adding a show endpoint. Now, for one thing, we don't always want to show all fields on our API endpoints, right?

All the fields on the model. That's something that we can get from modifying it in the ActiveModel Serializer, but what if you need different fields for different endpoints, like a show versus an index? Well, ActiveModel Serializer supports that through passing in flags like fields and includes for attributes and relationships respectively.

The thing is, I don't really like this design paradigm. I feel like this is too much putting view logic in your controller where I don't think it belongs. Not only that, it's really hard to keep consistent across different API endpoints. So what we did was we extended the ActiveModel Serializer,

which is just really easy in Rails. You just inherit from it. It's just like application controller. You can think about it the same way. We built an application serializer, which what it does for us is allow us to create views in ActiveModel Serializer itself. And so those would look like this.

Your serializer would now look like this. You have your standard attributes at the top, and those would be included every time that record is serialized. But now you have explicitly named view blocks where you can wrap attributes and relationships in those and call those views from any controller, just like this.

So now, with a one-line change and one serializer, you can now support two different views and even more so beyond that. It has really helped us be consistent, and one of the problems we were running into is that the solutions before this were creating multiple serializers, trying different serialization methods for different parts,

and this just makes it all consistent and all one. And that output would look like this. Nice and easy, very few lines of code that you need to mess with, and best of all, the developers who were using it didn't have to think much about how they were doing it. They just got to the part that they wanted to get to, which is, how do I get these attributes on this endpoint?

So, before we, another part of building APIs, and one of the things that I think is almost the most important is documenting the API, because I'm not going to mince words here.

An undocumented API is a worthless API. Documentation is like sex. When it's good, it's very good, but when it's bad, it's still better than none at all. Documenting your API, I cannot stress enough how important this is.

No matter how you do it, just do it. We started out documenting our API with wikis and markdown, and this is actually a great way to get your documentation spun up, because it's really easy. There's low barrier to entry. A lot of developers know how wikis work. They know markdown. It really gets everybody in the same place writing these APIs,

and it comes out pretty nice, right? Like, you know, the wikis and the markdown can really just work, but the problem is you go from this nice, well-maintained house of all of your nice-looking documentation, and then you sprinkle in a little time, and perhaps you add a new endpoint,

and somebody forgets to update the documentation. Perhaps there's a bug that they fix, and they don't even think about updating the documentation, and then you compound that across all of our applications and all of our tools, and you add time, and you quickly go from this nice, beautiful, well-maintained mansion to a dilapidated mess. So how do we prevent that?

Like, how do we not make that happen? Well, there's several ways, but the way that we chose to do it at Procore is with Swagger, AKA the open API specification. And if you're not aware what Swagger is, which a lot of people aren't, is it is basically just a specification

for describing RESTful APIs. In our case, we write our Swagger in YAML, but it supports YAML or JSON. Here's a JSON example straight from their website. It just basically describes all of your endpoints, all of their parameters, everything that is technical about your API endpoints.

And this gives us a lot of benefits. For one, it's a guided structure. We're at RailsConf. I'm assuming everybody here likes opinionated frameworks. If you're not, if you don't, you're probably in the wrong place. And Swagger is definitely an opinionated framework, but that helps a lot because it goes back to don't make your developers think.

They follow the Swagger spec, and they have nice APIs. Swagger also encourages you to separate your style and content. This is a paradigm we've learned, we've known in the web development world for a long time now, right? You don't want your documentation content and your style to be the same because if you'd update your style,

well, that's a huge endeavor, right? And this is just basically machine code and CSS. It's nice. One of the biggest benefits of Swagger is that it's machine readable because it's just JSON. So you can do things like test it, it's lintable. One of the coolest things you can do with Swagger is auto-generate SDKs.

It's true, there's a whole lot of tools that take the Swagger and do really cool things with it. I encourage you to check those out. It's kind of beyond the scope of this talk, but it is really neat. But that's not to say that it's all roses. There are problems with Swagger. One of the biggest is, well, it's another damn thing to learn, right?

It's not always easy. Not all developers know the JSON spec, so they have to learn that, and then they have to learn the specific parts of the Swagger spec, so there is a bit of a learning curve. But we've found that the benefits greatly outweighed the cost. So I want to talk about our Swagger at Procore, right?

Because remember, Procore is big. Our application is big. We've got a lot of things going on there. We follow a lot of the same paradigms writing documentation as we do writing the API, basically providing style guides, having reusable components as often as we can, and that's another great thing about Swagger

is you can reference other models within your Swagger specification, so you don't have to keep rewriting definitions all over the place. One of the problems that we ran into was that with all of our endpoints and all of our APIs, our Swagger was getting a little big, 7.7 megabytes big,

which for a JSON file is kind of huge. We basically broke every Swagger renderer that we could find. So with no other options, we basically wrote our own. This is an architecture diagram for our documentation generator.

It runs Ruby on AWS Lambda, which could be, I think, a talk unto itself, but it's kicked off by our continuous deployment and our continuous integration hooks, and it creates a static JSON that is ready to be renderable and read by humans, stores it on a CDN,

and then we have a React rendering application for our documentation site that pulls down that JSON. So what it means in the end is that we have nice, beautiful documentation that's really, really fast, and it goes across all of our endpoints, which, if I haven't said it yet, is huge.

So to kind of wrap up and talk about these aspects, don't make your developers think. If I haven't said that enough, it's true. You don't want them to have to make decisions

that they don't care about and they're not qualified to answer. Build a style guide for basically every endeavor, because this goes back to the same thing. If you have questions already answered, developers don't have to think about it. Disagree and commit. This is huge. We're going to get into arguments. We're smart, passionate developers.

Always be ready to essentially take the higher road and say, all right, we're going to go with your plan. Make contracts first. This is really important. There's a lot of benefits. I don't think I have to harp on that, because the first time you try it, you won't do it a different way after that.

Use reusable components. I think that's just a general programming paradigm in general, but it really comes into building APIs. Use reusable components everywhere that you can. And, for the love of God, document your damn API. Nobody wants to use an undocumented API. I mean, nobody.

Not even internal developers. So please, please document it. All right, again, my name is Derek Carter. I work at Procore. I love working there. We're hiring like crazy, and if you want views like this, please, please talk to us. We'd love to talk to you.

Thank you. That's a great question. So the question was, do we have basically internal routes that we don't want to expose to customers? Yes, lots. And the way that we handle that is one of the things that I didn't talk about in relation to the support level. Is that in our swagger, we have an ex-internal only flag.

And so the API developers can put that on their swagger documentation to mark the fact that that is an internal only route. And on our documentation rendering side, we use LaunchDarkly to essentially filter out those features that,

or those endpoints that are internal, and so that we only surface them to those who are basically approved to do so, like internal customers, and sometimes beta customers, but that's kind of rare. Oh, so the question was, is this something that's basically Procore specific? Right now, yes, it is, because it's actually pretty new in our system,

and it's kind of, it was experimental, but it's really, really working out for us. And so, like I said, this could be a whole other talk in itself, and I think maybe that's something that we can talk about. Because we really want to talk about how we've done things here

and really give back to the Rails community in terms of these kind of things. Speaking of which, also going back to the application serializer, that is going to be a PR into active model serializer, because I think that's a pretty good value add there. Yeah, so he was saying that JBuilder makes it hard to share partials across,

and we found that to be true as well. Well, active model serializer allows you to call other serializers either by name or implicitly by name, which is a huge value add to us. Oh, I see, I see. So the question was, if the endpoint can look differently, essentially,

how do we note that in the contracts? Well, very low level, it's JSON, we can add comments, basically, and that's really solved our problem. So the question was, the timeline on the filterable gem? Actually, I don't know, but this gentleman right here, I think, probably does.

Oh, oh, so I see what you're saying, the timeline for release. Okay, sorry, I apologize for putting you on the spot. Yeah, so the timeline for release, it's actually fully approved and it's ready to go. We just have to cross our T's and dot our I's and then it's out. Yeah, basically. That's a great question. The question is,

do we write our Swagger documentation by hand or do we generate it somehow from our controllers? Right now, it's all from hand. Just honestly, and the reason why is because across all of our applications, it's a very old application, not everything is consistent and so in reality, the best way to do it right now is by hand,

but we have explored lots of ways about how we're going to do that going forward. So the question was, how do we basically kind of gate keep the Swagger changes? Yeah, code review. It's all checked into our repo and all Swagger is code reviewed and we have tests and lint basically on our Swagger.

Well, thank you guys and enjoy the rest of your RailsConf.