Python and GraphQL
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 132 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/44998 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 20187 / 132
2
3
7
8
10
14
15
19
22
27
29
30
31
34
35
41
44
54
55
56
58
59
61
66
74
77
78
80
81
85
87
91
93
96
98
103
104
105
109
110
111
113
115
116
118
120
121
122
123
125
127
128
129
130
131
132
00:00
SoftwareFront and back endsCognitionQuery languageQuery languageLatent heatFormal languageEvent horizonSoftware maintenanceAxiom of choiceString (computer science)Parameter (computer programming)Endliche ModelltheorieQuicksortStructural loadMathematicsGraph (mathematics)Type theoryRevision controlCovering spaceQuery languageObject (grammar)Functional (mathematics)SequelMultiplication signExpandierender GraphFront and back endsDomain nameEvent horizonImplementationDependent and independent variablesSoftware frameworkShift operatorSoftware developerResultantCodeMereologyInformation retrievalJava appletPairwise comparisonPhysical systemLatent heatComputing platformRun time (program lifecycle phase)Wave packetFile formatStudent's t-testClient (computing)Context awarenessBitDebuggerFacebookTwitterEmailCountingField (computer science)Representational state transferFormal languageInternetworkingOpen sourcePay televisionBand matrixAddress spaceSound effectGroup actionServer (computing)Instance (computer science)NamespaceResolvent formalismData typeSoftwareDampingEuler anglesService (economics)System administratorOrder (biology)outputData structureArithmetic meanLine (geometry)Descriptive statisticsRoutingMiniDiscComputer fileSoftware testingOverhead (computing)DatabaseProcess (computing)RootBuildingCognitionFluid staticsMathematical analysisError messageEquivalence relationComputer animation
09:21
Core dumpRootEvent horizonInformationField (computer science)String (computer science)Electronic mailing listWindows RegistryQuery languageRootField (computer science)ImplementationSocial classType theoryJava appletNormal (geometry)Functional (mathematics)Library (computing)Standard deviationSystem administratorQuicksortModule (mathematics)Pattern languageElectronic signatureGraph (mathematics)CodeQuery languageCore dumpObject (grammar)Run time (program lifecycle phase)Computer configurationString (computer science)Graph (mathematics)AreaError messageMathematical analysisTelecommunicationAsynchronous Transfer ModeArtificial lifeEmailData typeScalar fieldForm (programming)Structural loadEvent horizonIntrusion detection systemComplete metric spaceProgramming paradigmRule of inferenceBoolean algebraData modelCoprocessorPoint (geometry)Fluid staticsBitRepresentation (politics)Uniform resource locatorVariable (mathematics)WritingSoftware developerTheoryBuildingMobile appWeb browserWindows RegistryCASE <Informatik>Right angleAbstractionMoment (mathematics)Process (computing)Ring (mathematics)Dependent and independent variablesInformationMessage passingIntegrated development environmentReal numberTwitterNumbering schemeData dictionaryResolvent formalismView (database)Arithmetic meanComputer clusterComputer animation
18:43
Front and back endsCognitionRootData typeEvent horizonHausdorff spaceError messageAnalytic continuationGraph (mathematics)CodeResolvent formalismArithmetic meanSoftware developerProduct (business)Run time (program lifecycle phase)Computing platformQuicksortDebuggerCartesian coordinate systemEvent horizonFront and back endsType theorySystem callCore dumpFunctional (mathematics)Endliche ModelltheorieSocial classProcess (computing)Game theoryReal numberUser interfaceSequelMultiplication signMereologyFocus (optics)Data storage deviceRoundness (object)Correspondence (mathematics)DatabaseStructural loadConstraint (mathematics)Revision controlInstance (computer science)Field (computer science)Moment (mathematics)Software maintenanceMathematicsCoefficient of determinationAbsolute valueQuery languageComputer animation
24:51
Event horizonLibrary (computing)MathematicsQuery languageMultiplication signLibrary catalogWritingDeclarative programmingFront and back endsLevel (video gaming)FacebookCodeImplementationGoodness of fitException handlingField (computer science)DebuggerQuicksortCASE <Informatik>TelecommunicationMappingDescriptive statisticsLatent heatTable (information)Object (grammar)Software developerClient (computing)Moment (mathematics)Analytic continuationInformation securityMiddlewarePoint (geometry)Formal languageDatabaseGraph (mathematics)Core dumpFlagTerm (mathematics)Modal logicType theoryMetadataJava appletLastteilungStructural loadServer (computing)Combinational logicComputer animation
Transcript: English(auto-generated)
00:04
Hello, my name is Alec and I'm a software developer here in Edinburgh I work for a company called administrate and we make a training platform for organizing and running your training business and One of Scotland's fastest growing tech companies. We have millions of students thousands of users and hundreds of customers use our platform
00:22
Around the world. We also work four days a week and we're hiring So give me a shout if you're interested in that, but I'm not here to talk about that. I'm here to talk about Oh APIs Hopefully, there we go So we're here to talk about why we rebuilt our rest api's with graph QL
00:41
And this isn't a rest bashing session or even a comparison between the two But I want to talk about some of the issues that we were having with our implementation of our rest api's And then how graph QL was the right choice and to help us tackle some of those issues So we'll cover what the basics of graph QL is How the tooling enables our development and then we'll dig a little deeper into how you can use Python with graph QL and
01:04
Specifically the way that we do it So here's an age-old battle between time and money and in a software development context that usually boils down to writing beautiful code or shipping features and when you're starting out the trade-off between quick results and money
01:23
Sort of makes sense you ship the feature Get the customer and stay afloat But that turns into technical debt pretty quickly Especially considering how quickly you can build api's in Python using the frameworks like Django or flask so
01:40
We had some issues with this administrator Some of our problems were that we had to put to we had two versions of a public API a private API and even though one of our Public API's was deprecated. We still had to support it keeping our documentation up-to-date was really hard We had a really tight coupling of the front end and the back end and to save ourselves some time in the beginning
02:04
We had automatically generated some some endpoints by sort of By introspecting our sequel alchemy models, which meant that we got a lot of functionality really quickly, but It was pretty difficult to work with in the front end The cognitive cognitive load was high for working with our with our API's
02:25
The name spacing was awful. I couldn't really find where anything is. Yeah, it's just it's just pretty confusing and There's basically just not enough thought giving to API's before we wrote them Our private API had tons of ad hoc roots in it
02:42
And it was sort of a get the data to the front end by any means sort of sort of attitude and So starting again with an API from scratch might sound a bit risky and it is but hopefully you'll understand why we chose to do This and agree with some of the reasons why after this talk so graph QL
03:02
What is it? It's an API specification And it's been built an open source by Facebook in 2015 and is now used by many companies as part of their API's GitHub PayPal Twitter and of course Facebook just to name-drop a couple but what actually is it? Let's have a look at the definition
03:20
It's a query language for API's and a runtime for fulfilling those queries with your existing data That's still fairly vague. But in practice, what does that mean? So as a developer, you're gonna have to define some data types in your code You need to create resolver functions to help you resolve those types against your existing data Then you pass the whole lot into a runtime and there's your API. That's pretty easy and still pretty vague
03:45
So let's let's dig in a little bit deeper Graph QL is not language specific. It's just a specification just an idea So that means you can implement graph QL in whatever language you like Good So that makes it really flexible to integrate with your systems
04:02
There is implementations in JavaScript Java Python go along you name it. There's probably an implementation The data retrieval part of it's also also up to you. You can use SQL alchemy models You can use Django models. You can use text files on disk. It's up to you Next up. There is a single endpoint so when you talk to a graph QL endpoint you need to build a query with
04:23
The types of data that you want and the fields on those types and then you post that off to the graph QL Endpoint and data is returned to you This makes data retrieval far more efficient It means you're not having to reach out to hundreds of services or many services to get the data that you need So it cuts down latency over the network immediately
04:43
Yeah, which is especially useful when bandwidth is at a premium like when you're doing mobile development or when the Internet's just slow The single endpoint also means there's no versioning pain and there's only one endpoint. So there's only one version So that cuts out the maintenance pain of maintaining two versions
05:02
You just keep adding types to your API and evolving it And for us to administrate was a really huge mindset change Going from having a private API where we could just literally throw in whatever we wanted and didn't matter To having a public API that meant everything that had to go in the everything that was going in the API It sort of had to be right
05:22
With graph QL you you get what you want So as you mentioned you create a query you send that query off and data is returned to you And there's nothing extraneous there The data that you ask for is the data that you receive Which means that you can you get a sort of efficiency on the back end to if the clients only asking for the data They need then you only have to fetch the data. They need from the database
05:43
Which means that you can make your queries more efficient Another part of this is the format. So when you define your query The order that you define your query in and send off to the server is the order the exact order that comes back in So what that does is it shifts the responsibility to the front end or the client who's making the query?
06:03
Shifts shift their shift the responsibility of getting the data to them and it really uncouples the front end and the back end From having to know what what data each one sort of needs. So Once once you have a schema defined then the front end knows exactly what data it can get and how it can get it
06:23
I just want to do a little illustration of this. So Here's a typical graph QL query it's a sort of a JSON like object we define Type on it. It's event type. We pass in a parameter event ID No more query strings, and then you've got some fields on their title
06:42
Title description and talks and then the response is going to come back as a JSON like object And it's in exactly the format. You've asked for it in now compare this to making Making a request to one of our API endpoints. This is our events endpoint and being you know a training platform
07:01
This is one of our biggest sort of biggest sort of domain entities You'd expect to get something back about events. We're not really sure what It could be anything. So this is an illustration of what comes back on our events endpoint fully expanded It's about 400 lines long. And yes, it's a little bit of a biased example because you know There's a lot of data attached to our events
07:22
But the main thing here is that there's no guarantee about what you're going to get back from your API when you're just querying And then an endpoint like that Processing a response like this in the front end is an overhead and it's also a risk So something in the back end could change and that could be a breaking change for the front end and you wouldn't know about
07:40
It until it was too late. Well, unless you had test I suppose. So the next thing I want to talk about is mutations Conventioned queries in GraphQL So if queries are the idempotent equivalent to making a get request on a rest API So queries should never change data on the back end Mutations on the other hand are the way that you would change data
08:03
So let's just take a look at the structure of that This is a typical mutation. Basically, you need the keyword mutation It's a JSON like object again You drill down into the type of mutation and that you want to run So here it's account and then on the account type we have the create mutation
08:21
That's not keywords. You can call those wherever you want, which is really useful for adding Workflows to your API. So for instance, we have a mutation called add learner to event, which is far more Which which is far more sort of? explanatory about what that action is going to have What effect that action is going to have in the back end instead of just sending a sort of a post request to?
08:44
An endpoint and sort of hoping for the best So once you've decided on the mutation that you need You need to add an input object With the fields that you want with the data that you want to pass into the system and then underneath that Where we we have account and then name and email address
09:01
that's a that's a query fragment and that's going to be the data that that the system sends back to us once we've run the mutation and As before it looks pretty similar. It comes back in the it comes back in the the format that we've asked for in So Next thing is that graph QL is strongly typed Benefits of typing for those who don't know is that you can do static code analysis to catch your errors early
09:27
Makes your code more self-documenting and you don't need to add comments to explain typing and it makes auto-completion better and IDs Explicit type definition is also quite a nice way to organize your API So every type that you have in your API needs to have a type in code
09:43
So that was much a much better way of organizing it then The way that we were organizing it before in our administrate And then the strong typing also allows some really powerful tools To be built against it and to aid the development for and the development of your API So Enough on the theory. Let's let's have a little go at building an API. So we're gonna follow roughly the process that we do
10:05
Administrate I'm not gonna do any live coding because that'd be terrifying. So first To make sure that to make sure that something that to make sure that what's going into the API is right We want to make sure it's all well planned out and thought out. So to do this we use a tool called graph QL Voyager
10:24
Graph QL Voyager is a tool that allows you to visually explore your graphical API as an interactive graph it's a great tool when you're discussing or designing your data model and It can be connected to your own graph QL endpoint Hopefully we look here so this is going to be our design
10:44
We're gonna have a root query type here. We're gonna have an event and we're gonna have a talk and once your graph QL schema gets more complicated you can see this would be a really great way of just Having a visual visual representation and making sure that things are working in the way that you you want them to work
11:02
so the process that administrate is that we need to have We need to have a design in code and review that design before it goes into well before anyone does any work on it So we have our design now. Let's look at some let's look at some code. There are a few options for Creating a graph QL schema in Python. The first is graph QL core
11:24
this is the core implementation of the Python the graph QL runtime in Python and You can create a type like this So here we're creating our event type use the graph QL object type It needs a name it needs a dictionary of fields those fields need types of themselves
11:42
You see we've got a sort of our graph QL scalar types there with a string and list And then we are just going to resolve the data there It's just going to be sort of simple case at the moment. This is just going to resolve some static data And once you have your type you can then Add it to our root query type that we saw from the design
12:01
It follows a similar pattern you need to have a name and some fields on there And then the type that we defined before event type that is going to be a field on this root query type And that's all wrapped in a graph QL schema And that generates the schema for you. So once you have your schema, how do we make this into an API? Well, we can use
12:20
We can use the flask graph QL package and basically just initialize your flask app you add a URL rule Use this graph QL view Object from from the flask graph QL library and then you pass in your schema and that's it that's as easy as it is to get going but That's kind of messy, isn't it? That's gonna get messy really quickly
12:42
So we can do better than that. So next in the ring. We've got graphene Now graphene is the most popular sort of abstraction of the graph QL core implementations by get up stars And and here's the reason why it allows you to write graph QL schema in Python and much more Pythonic way here
13:02
We've got a Python class got some static variables on it Those are our fields and then for the slightly more harder to well once If it's a scalar field, then graphene will just resolve that data type for you But if you need to resolve something a little bit more complicated you need to define a resolver on that class
13:20
So you can see we're just resolving our talk standpoint there And once you have those it's a similar pattern again You need to add those to your root query type Which basically just collects all all the types in your API and then you generate a schema by using graphing that schema And once every scheme object is passed out straight back into the the flask and the flask graph QL
13:43
Packaged here. Now. We decided not to use graphene that administrate Because we like to make things hard for ourselves. So we decided to roll our own So graph QL eyes is the name of the library that we are Writing to help us with our graph QL API
14:01
Why why do we do this? Graphing allows us to write graph QL schema in a more Pythonic way But we asked ourselves Why should have to write graph QL schema at all We took the inspiration from Java's implementation of graph QL and Java being strongly typed Allowed schema to be built through introspection
14:23
So you pass your pass your types into that and it generates a schema for you And we find this is possible to do using Python threes type annotations So, I mean type annotation isn't real typing but it does allow you to pass extra information when you create your Python classes So we consume those type annotations by introspecting the types that we declare and then we build a graph QL schema for you
14:46
The benefits of this are that our developers don't have to learn any new paradigms You don't have to learn and graphene or or anything like that. We just write pure Python and and that's it So the cognitive load is much lower when you're when you're doing this and it also makes the implementation easy to change out
15:02
So when the new graph QL comes out in a couple of years We'll just be able to swap that out and use the classes that we've already written. So what does this look like? Well looks like a couple of normal Python classes and you just have to make sure you've got your type annotations there and once you and the the methods on those classes are are going to be the fields of those types and then
15:23
Executing those methods is going to be the way that we resolve the data And once you have that and you need to add them to your root query type again It's the same same kind of pattern but with graph QL eyes you need to initialize a registry And then add the query type Run the query type through the registry and then generate a schema and that graph QL schema object is from a graph QL core library
15:47
And once you have your schema, you can just use that in your your flask and graph QL package again Can you use it not yet, I'm sorry about that It's a real bait and switch
16:01
Turns out open sourcing stuff is hard and we want to do it, right? So if you are interested just follow me on Twitter, I'll post when I'll post when it's released or you can sign up. We've got a type form there We'll send you an email if you prefer that that mode of communication, but yeah, sorry about that in the meantime The stuff that we're relying on heavily is the inspect and typing modules of the Python standard library
16:26
We use functions like signature and get members from inspect to help us with their introspection And if you have not already seen the typing module in the standard library It's worth looking at too It provides you with all the Python types to do annotations with and it's got some useful helper functions like get type hints
16:42
And to help you do type processing on receiving annotated Python objects So we have got our design we've written some code what's next we need to make sure that it's working I'd like to introduce another tool to you if you've not already seen this is called graphical
17:01
and graphical is basically an in-browser IDE for exploring your It's basically an in-browser IDE for exploring your your graphical API what we can do here is
17:22
Actually some there's a little bit of live coding. I suppose you can start your You can start your query there. You can see that it's got some nice auto completion features It's Gonna be a one
17:43
and once you've created your query you can run it and
18:00
You can get the response from your graphical API sort of right in the browser there. It's a really powerful tool for for development It gives you great visibility over what's happening when you're building your API Another thing I didn't mention before is that Because graphic URL is strongly typed it means that it can generate your documentation for you. So
18:22
In graphical here, you can see that we've got the documentation for our little toy API here You can drill down into the types and write down into the scalar types there So it makes it really easy for for keeping your your API documentation up to date
18:47
Right API first given the game away. So what all these tools help us do at administrate? It helps us think develop with API first in mind and For those of you who don't know what API first is well, it's fairly self-explanatory, but it sort of means that
19:04
We develop our our product as well. Our product is our API. So this is the thing we think about first This is thing that's that's sort of most important to us You might say that your product is your web interface or your mobile application But
19:20
for us administrate we're building a platform which means that we need to we need our Customers and third parties to be able to integrate with us really easily. So our API is our is our product What does that mean in practice? Well, we need to have some upfront design As as you can see we did that with we did we did that with graph QL Voyager
19:43
It just means that we are able to to decide how our API is going to work before Anything's built on it We also release publicly and regularly Well quite regularly and we're moving towards a continuous delivery type type situation, but the moment it's every week
20:01
and when we release There's nothing feature flagged in our API. So when a field goes in the API and it's released it's you know public to the world and This is a really nice constraint sort of focus our attention on delivering and good value incrementally every week Of course there are times when we need to deprecate something something, you know stuff does go wrong
20:25
But graph QL's got us covered there deprecations built into the schema And also there's benefit of there being one Endpoint means that every request goes through your graph QL endpoint Which means that you can track what what types are being requested for and even who's requesting them
20:42
So there's going to be no more pain about wondering If deprecating a field is going to be a real problem for somebody if you can see that it's not been used for the last six months next up dog fooding and for these for dog fooding is the process of Using your own product. So we consume our own API to build our own features
21:01
And this is really great for for understanding the the developer Developers of pains and well the developer experience when developing against your product So if there's if there's a problem with your developer process Then if you're using your own API, then you're gonna be the first to spot it and documentation
21:22
It's a real pain. Well, we had a real pain with our documentation being a different process from our code and Our sort of our code process and now the two are completely Are completely sort of merged together. So documentation changes can be can follow the same process as Code review so you can it never it's much harder for it to get out of date
21:45
Than before so let's go back to the problems that we were having Maintainability, there's just one API version Our documentation is easy to keep updated the coupling of the front end and the back end well, that's completely separate now because the back of the front end is is
22:02
Is completely needs to well the front end needs to know what what data it's going to get A cognitive load is less. We're just build. We're just writing pure Python classes And not enough thought was given to our API well That's the first thing we think about and when we wake up in the last thing you think about when we go to sleep So I think that's about it. Thank you very much for listening and any questions
22:52
And so so it it takes a guess that well, yes So it generates it from the schema and it can sort of tell you what types are in it But there's also the ability to add your own documentation, which I didn't show there
23:04
But you can you can sort of provide documentation as you're writing the types as well So it can be so GraphQL takes you part of the way and then you know, you can you can add your own as well
23:29
Well, it will just you sort of just get runtime error and it's up it's up to you to handle them So you can handle the errors yourself. I mean it is just It's whatever the errors from GraphQL core. It'll be probably pretty nasty if you've not you've not handled it properly
23:44
But yeah So I know next to nothing about GraphQL But you've shown the this flask application that you know Takes the graph graph to a Cree and around runs it against something somehow. Yeah, but like whoa, how does it do that?
24:05
Does it do it for like thank Johnny to an SQL data store? Yeah. Yeah, absolutely. Sorry I maybe wasn't wasn't clear on that one so when you define in the types that we saw and there were resolver functions and
24:22
Maybe we can go back to them in your resolvers. We were just as the example Say for instance here This was just returning some static data, but that could easily just be a call off to your SQL alchemy database with the corresponding type
24:42
So say you have a SQL alchemy model event and that has you know a name on it And then that that resolver function there could just be a call to get that Particular so I just do the mapping myself. Yes. Yeah. Okay. Thanks Hi, hello
25:01
So you say before that in case of deprecation GraphQL got you covered Yeah, I don't know nothing about GraphQL except your talk. So can you elaborate on that, please? Yeah, so the the Library like graphene allows you to mark things as deprecated and when the so there's a few front-end libraries as well, which I didn't sort of get into but they're they're able to
25:26
Introspect the your GraphQL schema and they can see which things are deprecated and then in your documentation or your your your sort of Your developer portal and you'd be able to sort of see these things come up as deprecated. So It's supported by the Python libraries that we have available to us and so you can you can sort of mark them as deprecated
25:45
but Also, I mean there's this. Yeah. Yeah. Thanks. Cheers Thanks for the talk And maybe it was kind of because because you don't they wanted to talk to be very lengthy But did you have any issues with pagination?
26:04
Yes, you have to write You have to do it yourself But it's yeah, you it's just something you need to you need to implement yourself think I mean same with I get Yeah, same with sort of pure rest stuff Sorry, I can't be I
26:27
Wonder in the world of graph QL, where did you store? Where would you store? There is some combination about Metadata in terms of fields description
26:46
You could Is So what what
27:07
So the library here that you so basically this documentation is generated from an introspection of the schema that you've declared so Every time you make a change to the schema in the back end
27:20
the library in the front end Will introspect the schema and the documentation will be updated automatically, so you don't need to store it in the database Documentation can be added to your types I think in In graphene anyway, I think you can provide If you there's sort of a there's a sort of a description
27:41
Flag that you can that you can add here So some of it will be in your code and then some of its automatically generated by whatever is introspecting your your schema Does that answer answer your question? Sorry
28:04
Thank you for your talk How do you do you deal with some form of like either like malicious queries? Like how do you deal with somebody basically requesting a full database dump or Tons of queries that cause a lot of load on the back end on your data store. Yeah, I guess
28:24
We would try and deal with Well, we would try so we would try and deal with stuff like that maybe before ever got to GraphQL So we try and maybe deal with that in our load balancer or nginx or something like that and so ideally we would try not to let that stuff kind of
28:41
Get to a point where it was executed. So You can also you can add custom stuff because it is one endpoint You've got a bunch of middleware to do whatever kind of security checks that you want to do on requests coming in So it's yes, it's up to you, I guess Thank you
29:00
Hi, you said you're moving towards a continuous deployment sort of world at the moment. Yeah We do this. We also use Groff QL and one of the issues that we're Seeing in the future is a place where our clients and our server have different ideas of what the schema is I was just wondering if you had any comments on that any tooling that you knew about or any way of sort of
29:25
Checking those things before That's a good question. And I guess we've not we've not quite got there yet there I mean, I think there are there are things like the relay specification which Sort of allow Automatic which sort of allow communication between the front end and the back end
29:42
That's a react library and sort of out of the scope of my of my expertise But yeah, I guess you I'm not sure how you deal with that We've not we've not quite got there yet But let me know if you come up with anything good because it sounds like that's gonna be a problem. We'll have in the future
30:04
one last one have you a lot of support in JavaScript and what was the were the pros and cons so and yeah, so the
30:20
Original graph QL implementation was in JavaScript and that was you know, that that's maintained by Facebook the reason that we're doing it in Python is a long and sad story about how we try to write something in Java and Nobody wanted to do that And so it was out of necessity that we we sort of ported it to Python
30:40
Which is sort of the main language that we're using at administrate and but we find that made people a lot happier So yes developer happiness is the main main reason for that