​Rails APIs: The Next Generation

Video in TIB AV-Portal: ​Rails APIs: The Next Generation

Formal Metadata

​Rails APIs: The Next Generation
Title of Series
Part Number
Number of Parts
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
This is a sponsored talk by Procore. Building a consistent API for a large and long-running monolithic API on a tool-segmented engineering team can sometimes feel like herding cats. REST, serializers, and Swagger: Oh my! Learn what worked (and didn’t!) as we go behind the scenes building Procore’s first open API.
Data mining Computer animation Software Web-Designer Software developer Projective plane Bit Office suite Line (geometry) Cartesian coordinate system
Game controller Computer animation Direction (geometry) Counting Energy level Endliche Modelltheorie Figurate number Musical ensemble Cartesian coordinate system
Ocean current Game controller Decision theory Direction (geometry) Debugger Virtual machine Bit Database Mass Mereology Cartesian coordinate system Product (business) Data management Computer animation Causality Endliche Modelltheorie Computer-assisted translation
Computer animation Bit Resultant
Computer animation Code Natural number Software developer View (database) Software architecture Flow separation
Mathematics Computer animation Multiplication sign Analogy Debugger Software framework Shape (magazine) Physical system
Computer animation Gender Software developer Cartesian coordinate system Computing platform Product (business) Physical system
Computer animation Angle Bit Cartesian coordinate system Entire function Computing platform
Point (geometry) Code Software developer Multiplication sign View (database) Control flow Representational state transfer Parameter (computer programming) Revision control Mathematics Computer animation Personal digital assistant Different (Kate Ryan album) Video game Right angle Software architecture Boiling point
Latent heat Process (computing) Computer animation Software developer Decision theory Core dump Electronic program guide Bit Endliche Modelltheorie System call Product (business)
Standard deviation Code INTEGRAL Decision theory Software developer Multiplication sign Boilerplate (text) Source code Electronic program guide Sheaf (mathematics) Representational state transfer Cartesian coordinate system Product (business) Wiki Data management Process (computing) Computer animation Different (Kate Ryan album) Configuration space Right angle
Mobile Web Computer file INTEGRAL Software developer Decision theory Electronic mailing list Design by contract Cartesian coordinate system Power (physics) Attribute grammar Computer animation Personal digital assistant Software testing Data conversion
Logical constant Complex (psychology) Building Group action Beta function Serial port Code Multiplication sign View (database) Design by contract Function (mathematics) Mereology Perspective (visual) Computer programming Fluid statics Mathematics Different (Kate Ryan album) Bus (computing) Flag Cuboid Endliche Modelltheorie Error message Programming paradigm Block (periodic table) Software developer Feedback Electronic mailing list Sampling (statistics) Bit Instance (computer science) Type theory Process (computing) Internet service provider Self-organization Right angle Data type Row (database) Point (geometry) Filter <Stochastik> Game controller Open source Computer file Civil engineering Connectivity (graph theory) Field (computer science) Product (business) Attribute grammar Causality Term (mathematics) Energy level Software testing Associative property Alpha (investment) Consistency Interface (computing) Line (geometry) Cartesian coordinate system Subject indexing Uniform resource locator Computer animation Personal digital assistant
Wiki Word Goodness of fit Vapor barrier Computer animation Software developer Stress (mechanics) Right angle Parameter (computer programming)
Mapping Multiplication sign Core dump Cartesian coordinate system
Latent heat Computer animation Personal digital assistant Software developer Website Software framework Parameter (computer programming) Data structure Representational state transfer Open set
Web 2.0 Programming paradigm Machine code Computer animation Personal digital assistant Multiplication sign Weight Content (media) Virtual machine Right angle Special unitary group Resultant
Point (geometry) Curve Programming paradigm Software developer Connectivity (graph theory) Electronic program guide Bit Special unitary group Mereology Cartesian coordinate system Latent heat Computer animation Right angle Endliche Modelltheorie
Computer animation Computer file Computer configuration Volumenvisualisierung Data storage device Set (mathematics) Website Bit Diagram Software architecture
Point (geometry) Game controller Context awareness Serial port Connectivity (graph theory) Decision theory Multiplication sign Real number Electronic program guide Design by contract Parameter (computer programming) Mathematics Latent heat Meeting/Interview Term (mathematics) Energy level Software testing Endliche Modelltheorie Partial derivative Physical system Programming paradigm Theory of relativity Mapping Software developer Planning Representational state transfer Cartesian coordinate system System call Computer animation Right angle Routing
V a and here a of solar the would introduce
myself so just get the toned looks up on stage of my name is Derek Carter of my work for Brokaw I've been a web developer for about 20 years and then I better rail developer since 2008 the end of this picture was taken was taken by a colleague of mine and is from ah office I kid you not I love our opposite they go walk out here every single afternoon but but for some of you are cut john's talk yesterday you know a little bit about the history of pro-poor about what we are is cloud-based construction management software we as a tag line says that I really love we build the software that builds the world end up I'm up here today to tell you guys about an endeavor that recorded in 2016 and on into this year which is the proper decided to really doubled down and build out a complete API across all of our projects in our applications and tools and to give you a sense of what that endeavor entailed I wanna talk a little bit about the gets a broke off for a 2nd but
Brokaw has were growing like crazy the last count I think we have over 700 employees and about 120 engineers probably more than that and at least 20 2 squads these troubled figures our real that is over 10 years old we have over 40 distinct tools in each tool is basically an application unto itself and we have over 500 controllers and applications Brokaw is big are application is big it's all a real that it's it's you'd forget majestic model it we are are majestic globally but she thought to be a
great name for a metal band of but beyond that we we operate with squads and 0 want something the Brokaw really believes then is autonomy not just the squabble with at the personal level we really believe that we should give direction to people and let them use their experience and their knowledge to get us where we need to go so we we strongly believe in
that idea of autonomy would something with that we tell to every new hire it's a strong part of our current of our culture and some we believe very strongly in and so the talk a little bit about that Tommy we follow
the Spotify squad model essentially we break up our our R&D department In 2 squads that each have a product manager UX designer and and a handful of engineers and the squads have ownership of a certain tools and that they have basically full control over they have a product ownership over they they choose the direction of the product that you want to work on next and it's that Tommy idea but and uh if you guys are familiar this squad model squads roll up into tribes and then for the things that go across squads for common interests I we roll up into guilds an example of a few that we have a pro cause we have a front end guilt that takes on a lot of the front and concerns of how we make decisions their performance skills that I keeps an eye on our performance and Howard doing an hour controllers in our database and we haven't even have go for massive failures with applications lodged we have we have a lot of air interface just from machine hookups and so we have this model and that has with at least 20 2 squads right each owning their own tools on a big application and we have to make a consistent effort to make 1 large API across the entire application how we do that without feeling like were herding cats
well it wasn't always a smooth and perfect ride but we got there and I'll show you a spoiler this is the end
result we've got a full consistent of beautiful API I think it's beautiful but you know but you know kind of like a mom mom here but of the we we we essentially accomplished what we set out to do but before I get into how we did that on talk a little bit about why
we did that why did pro caught a double down and best buy in an API with a lot of benefits to building an API like this of both internal which relates to your developer happiness code health and also external which is you know relating to our customer happiness and year sales health 1 internal benefit that you think
get is building an API forces a natural separation of concerns you now have essentially your API layer separate from your view later and that of course is a modular architecture and the modular architecture is innately cleaner architecture of
another internal benefit that we get
is his change tolerance if we get as a DJ should think says a new Rails framework every 3 months but if you build a a good consistent are well-designed API it can last through several iterations of different the JavaScript frameworks are front ends you can replace that any time we can integrate with any system in your API remains relatively the same I like the analogy of a TV we've had televisions in their living rooms for almost a century now and the API for how a television gets its powers and show the power plug has remained the same even though our televisions have drastically changed shapes and so when you design it well you can you can last for a while but 1 big external benefit is that the PR is
gender customer trust if a customer is looking at different products it looks in your product and has a well-maintained API it makes them comfortable knowing they can get the data in and out of your system easily that's gonna really help close the deal is really help engender that trust between you in the customer before they've even engage you another is that the API can
greatly expand your capabilities when you empower other people integrated developers reduced people were interested in your business base to build on your platform you can it greatly expand what your application do without spending any extra resources and when that works and that works well I think it works well approacK or you get to become the ecosystem you
now have fostered an entire ecosystem of applications that work on your platform and and everything just works nice and you get to enjoy and reap the benefits of that so I talk about the benefits of such a little bit
about what makes a good API we know when we're starting this endeavor was kind of the angle that we wanna look for well a good API
is predictable and consistent developers don't want to have to write new code for every n point you surface you are an especially this compound the fact when you think about developing st case reapplication them more in inconsistent points are the more code in the more Croft is going be introduced trying to take on those endeavors a good API is static if you're API changes in a breaking way congratulations you now just broken everything that is integrating with the the or who 1 joke that the haiku like is writing an API is like sex make 1 mistake in your stuck supporting at the rest of your life they no 1 of a good a good API is also a simple and clear and writing an API is not the time to be clever or show how good you are at writing code an API should get back exactly what somebody expected to nothing more and nothing less a good API is also flexible and this may seem right at 1st glance that it conflicts with the 2nd point that how can be both flexible static but think about steel right is the way you make steel not break it make it flexible and that's the same thing with an API so we don't know what a good API should look like those talk about how we
actually did it how did we build this big beautiful vs well it was easy it was always easy and was always pretty there was a lot of discussion there was a lot of arguments we have a lot of really smart and talented engineers approacK or and and some of them have different ways and different views on how to do things so there was a lot of arguments is a lot of style discussion there's a lot of architecture discussions there's a lot of conflicting ideas a lot of like really passionate arguments but 1 thing that a boil down to something that Jeff Bezos so that I really buy into is disagreeing commit you're
always going to have those kind of things that it seems like an impact but some just gonna give you just gotta for the sake of productivity decide OK we're gonna go with this I don't think it's the right way I believe it's this way but I'm gonna commit to this decision and be behind us 100 per cent on this so I wanna step back a 2nd it's a little bit about what
we did to the squad model appro core a few things that new endeavors they were doing that really help this endeavor and when things without it is the idea of what I like to call guides squats and against what is essentially squad that doesn't necessarily its own tools but really owns the process or other things that stretch across other squads it's like a guild but with more ownership of for example I'm on the API squad appro core and whereas I don't write a lot of the endpoints of those are owned by the developers who own the specific tools but we act as kind of a shepherd too the API development process by helping out you know to make people disagreeing committee and to come up with those decisions that are really important to keeping up our productivity so I was a meal and when we do as a squad how we help this endeavor of all 1 as Steve Krug famously says don't make your
developers think on things that don't matter to them that's a big 1 are 1 of the biggest questions that I get a Brokaw is of you know what should I do this or this and that it doesn't matter to them they just wanna know how to be consistent with the rest the application us so we wanna make sure that they're not having to think about these problems so for things they don't care about so they can get to the problem that they do care about like the tools and their endpoints assume 1 way is 1 of the ways that we solve this is to have a style guide it goes back to the rails concept of convention over configuration and we we broke or take this and that and we built a style guide for API we have standards for a lot of our different sections or application and now I wanna make sure that you know that you have to have a style guide planned out all advances matter of fact I think that's probably a very bad idea because you don't know where those decisions are all at once underway encourage you to do is I can keep a wiki or some kind of way that every time a question that comes up and you answer if you write it down it's the very important that you write down because you can't come to a decision and walk away the next day how many times like a when they write it down and then after a while guess what you got a style guide of ours looks kind like this this is just a small section of it but we we like to keep things as pretty as we can that so it's kind of formatted of but we just this really really helps especially for new developers who get on our application and start writing API is to have a lot of questions answered before they have to go and talk to another developed another key thing is boilerplate examples because let's be honest when you're when we're writing code were copying a lot already anyway right so as my eyes well make sure that we get the sources close to what we want the end to look like is possible if we know what that should be
so so is this scenario that of of the you guys may run into we did a lot of the that developer writing an API so he started Haraway writing that API and then you got your front developer whose that there's waiting on a diet this is give your mobile team you to year integrators whatever and finally OK he's done with the API ships that over all right we got it we can start working many realize is that no no it doesn't have half the things I really need i and also the back and a go back using camera maybe that all those the scenarios and induced by to the front of altering the brand wellbore celebrates into the product manager goes of only so we have customers who really needed to do this this and this and you back to right through to the beginning again of so how do you had is the process how do you get rid of that because you know what people he what developers ever waiting right you would be productive well 1 of the big things that we do is is really
contracts API contracts and it for those who don't know a API contract is just essentially an agreement between your stakeholders about what that API is going to look like when it's finished up in our case we just use a song files have an example would be something like
this on this before a example to do list application which we love the rails community right up and we just we pass these around and we get agreement and by and and the the front developers your mobile developers integrators and look at this and go OK this has all the attributes I need or I need this this nasty things you cut through the conversation a lot faster and not only that well when you made that decision in you've all committed to the central contract then your from a developers can start developing against it while you back in developers start writing tests against it and you they develop and they come to the middle and the exact they know exactly what the middle looks like and it works in in in powers that parallel development so I would like
to talk a little bit about the building static API is flexibly and because we found that there are there are a lot of times where we were building points that didn't actually meet the customer's needs and sometimes you can't get contracts out to that right sometimes it's not feasible to figure out all the things you can point needs to do you have a time so 1 of the things that we do a pro cause that we introduce the idea of support levels for API essentially we call them alpha beta and production and a lot of times if the developers writing an API for a tool but he's not sure that it meets all the needs of the stakeholders and is not able to write a contract for worth but he will make mark that is data and the internal customers or the actual customers who want to jump on these data programs demand with the full knowledge that this could change in these are kind of up in the air and then they provide that feedback right that that we really want for a product perspective and what that does is that takes that process which is instead of just like putting of steel ball more like pouring concrete we have a little time to massage it but after poured before it sets and that's essentially what we do and once we have them all at the back then we promoted to production and we've got our new point another important 0 . I
have a dive into this is use reusable components there is it is impossible in my opinion to build a consistent application without building some common components so let's go back to our to-do list up right this is just a simple index action for a single to do list to return the items but let's say we now have a of product requirements that we need to be able to filter this point to whether the items are completed well maybe look something like this right and that's why code where were you know were stepping through this but it's not the prettiest right but this is this could happen but when I think about this and this is just 1 filter on 1 end point we have over 500 controllers application we have a lot of tools very big suddenly you're in your code your application is now just full across and it's everywhere so how can we solve that well we built a genome that we call cultural and we include that on ah controllers and it's a really simple really nice interface I just include the concern and then you can say filter on your you and you can give a type to validate against us and I have built a validations for free as well but this means this is much clearer code much more readable and and that removes all the complicated bits to in a central location that's that you can provide full testing error and not only that but most importantly all of your points across the entire application now filter exactly the same way so all I see you gained all of that consistency for very little cost but we've also expanded it to support sorting and and it also provides data types as well as scopes if you need more complex filters and I'm happy to say that we're going to be releasing this gym as open source to the rails community of so let's have another very important part of developing API eyes is serialization in my experience has been 1 of the biggest performance costs pretty eyes on it's a constant problem that's where a lot of people and new to an API development really struggle with and we found that we were using J. builder for a while but i j builder is very flexible but it's not that it didn't have the performance requirements that we needed and we found that it didn't really serve are are needs in terms of organization so we went with the actor model serializer which we found to be up for our instance of better organized a bit more accessible and faster and for those of you who are unaware well serializes is a gem supported natively in the rails application of you just gym file and you can implement a serializer just like this you are explicitly list under atom and it also supports relationships and those relationships have serializes it supports those as well right out of the box and the beauty of this is because it's support rails you don't even have to change a controller code it sees a uh serializer with the same name as your model will automatically use that and you get the benefits of it right away a sample output for a that's religious or would be this and you can see the attributes there and you can see that included the association but it's just it's nice it works on the bus I let's add another and . 2 R API in this case were adding a show in place now for 1 thing and we don't always want to show all the fields on on our API points right all of the of the model at something that we can get from modifying it in the Act model serializer but what if you need different fields for different points like a show verses index while accuracy Aliza supports that through passing and flags like fields and includes for attribution relationships respectively but the thing is I don't really like this design paradigm I I I feel like this is too much of putting Viewlogic in your controller where right don't think it belongs not only that it's really hard to keep consistent across different API endpoints so what we did was we extended the ACT model serializer which is really easy and rails you just inherit from it it's just like your application controller because there the same way I we built an application serializer which but what it does for us is allow us to create use in model serializer itself and so those would look like this year your serializer would now look like this used had your standard edge at the top and those would be included every time that a record is serialized but now you have explicitly named new blocks we can wrap up attributes and relationships in those and up call those views from anywhere from any controller the just like this so now with 1 line change and 1 serializer are you can now support 2 different views and even more so be on that but it has really helped us be consistent and other 1 the problem we run into is that the the solutions before this work creating multiple serializes I'm trying different civilization others for different parts and this just makes it all consistent and all 1 and that output would be a and look like this nice and easy of very few lines of code that you need to mess with and best of all the developers who were using it didn't have to think much about how they're doing they just got to the part that they wanted to get to which is how I get these attributes on this point so that before we and there's another part of of Billy again as in 1 of the things that I think is almost the most important is documenting the
API because I'm not arguments words here and undocumented API is a
worthless API that I documentation is like sex when it's good it's very good but when it's bad it's still better than none at all yeah the document here I cannot stress enough how important this is no matter how you do it just do it how we started out documenting our API with wikis and mark down in this is actually a great way to get you documentation spun up because it's really easy there's low barrier to entry a lot of developers know how wikis work how they know markdown down I really get everybody in the same place writing these API it looks like it comes out pretty nice right like the you know the wikis in the markdown can really just work but the problem is you go from this nice well
maintained House of all of you nice-looking documentation and sprinkle a a little time and perhaps you have a new endpoint in somebody forgot free updated documentation perhaps of the body they fix and they don't even think about updating the documentation and the new compound that across all of our applications in all of our tools and you had time quickly go from this nice beautiful well-maintained mention to a
dilapidated maps so how he can we prevent that I can we not make that happen well there's always some ways but the way that we chose to do with a pro core is with swagger of
a the open API specifications and and if you know where swagger ends which a lot of people aren't is that it is basically just a specification for describing our RESTful API but in our case so it's we we write or a swagger in general but it supports Himmler Jason and here's a Jason example straight from the website of it it just basically describes all their endpoints all the parameters everything that is technical about your and API endpoints this gives us a lot of benefits 1 is a guided structure
where whales come from assuming everybody here likes opinionated frameworks but if not you know the problem the wrong place that is what is definitely an opinionated framework where that helps a lot because it goes back to you don't make your developers think they follow this while respect and all and they have nice API a swagger also encourages you
to separate your style and content this is a paradigm we learned that we know and web Bellman world for a long time now right you don't want documentation content and your style the same because if you don't they just the hell well that's a huge endeavor right and this is just basically machine code in CSS but it's nice I 1 these
benefits the swagger is that it's machine readable because it's just J Sun see you can do things like it's that tested its weights of all but 1 of the coolest things you can do a swagger is auto-generate st case but it's true there's all whole on tools are that that take a swagger do really cool things with that I encourage you to check results can be on the scope of this talk but it is really that's not to say that it's all roses there are problems where 1 of the biggest is well another damn thing to
learn right it's it's not always easy not all developers know the J Sun speck of so they have to learn that and then they have to learn with the specific parts of the swagger spec so there is a bit of a learning curve but we found that the that the benefits will greatly outweigh the costs so I won't talk about but our swagger a
pro-poor right because remember pro is big our application is big without a lot of things going on there I want you know we follow a lot of the same paradigms writing documentation as we do writing API basically providing style guides and having reusable components as often we can as another great thing about Swaggers you can reference other models of with the swaggered specification so you don't have to keep rewriting definitions all over the place 1 of the problems that we ran into was that with all of our and points in all our API is our swagger was getting a little
bit of 7 . 7 megabytes big which for adjacent file is kind a huge I we basically broke every and swagger renderer that we could find so with no other options we basically road our own
the this is a i an architecture diagram for our documentation generator of it runs on Ruby on a dubious land which could be I think a talking to itself but the but it's it's kicked off by are a continuous deployment of an inducer gration books and it creates a static Jason that is ready to be renewable and read by humans stores it on a SET N and then are we have a reactor rendering application that for our documentation site the pull down and itself so what means in the in that we
have nice beautiful documentation that's really really fast and it's uh goes across all of our imports points which by haven't said yet this huge so that our data wrap up and talk about these aspects of this don't make you developers thing but is that enough it's true you don't want them to have to make decisions that they're not really they they don't care about and they're not qualified answer build a style guide for basically every endeavor because this goes back to the same thing if you have questions already answered developers don't have to think about it this agreeing commit this is huge for any given arguments were smart passionate developers be always be ready to essentially take the high road and say all right we're gonna go with the plan up make contracts 1st this is really important is there's a lot of benefits I don't think at the hop on that because the 1st time you try it you won't do it a different way after the use reusable components I think this is a general programming paradigm in general but it really comes into the layer guys use reusable components everywhere you and all of that document and MAP I know nobody wants to use an undocumented API I mean nobody not even internal developers so please please document
all but again my name is Derek Carter iWork approacK or I love working there were hiring like crazy and if you want to abuse like this please please talk to us and we love we love to talk to you thank you that's as so your great question so the question was do we have a base in the internal routs that we don't want to expose a customer's yes lots the way that we handle that is and 1 of things that in talk about in relation to the support levels is that in our swagger we have an X internal only flesh and so the API developers can put that on their swagger documentation to mark the fact that is an internal only the root and on our degradation rendering side we use a lot starkly to essentially the filter out those are the features that are those endpoints that are internal and so that we only surface them to those who were basically approved to do so like internal customers and sometimes and you know comes to customers but that's kind of rare 0 so the question was is this something that's bassist Brokaw specific about right now yes i it is because of it's it's actually pretty new and in in our system and I and it's kind of a you new it was experimental but it's really really working out for us and so legs that this could be a whole other talk in itself as they may be of that so that we can talk about because you know we we really want to talk about how how we done things here and really get back to the rails community in terms of these kind of things of speaker which also go back to the applications utilizes that is going to be a PR into actor model serializer on because I think that's of pretty good value at the end of this so he saying that the makes it hard to share partials across we found that to be true as well and as an well Act model serializer allows you to call other serializes either by name or implicitly by name which is a huge value at i see i see so the question was if I had point can look differently and essentially I had we had we note that in the context of well a very low level it's J. we can add comments basically and that's really us all the problem and so the question was the time when the filter holder but actually I don't know that this this jomon right here I think probably does and all of those using the time of her release OK so you have that and I apologize Rabbani others but I guess at the time of release but it's actually of all approved and is ready to go we just have to like cross at is not arise in this in this that's that's a great question the the question is how do we write our swagger documentation by hand or do we generated somehow from ah controllers right now it's all from here just honestly and the reason why is because are are all across all rock real it's a very that patients that everything is consistent and so is really in reality the best way to the best way to do it right now by hand but we have explored lots of ways of how we're going to they're going forward yeah so and so the question is how do we had we basically a candidate gatekeepers the swagger changes of yeah could have you know it's all check into a repo and and all swagger is could we have and we have tests inland basically honest way with the has enjoy the rest of your got to have it so
if I found if you were the