Designing a Pythonic Interface

good morning but please welcome our next speaker of on the ground will be talking about designing and by tonic interface thank you hello

and good morning president

said Meyer mining using technology like to talk

to you about designing iconic interfaces and and very high probability for for my talk which only occurred to me later is an illustrated guide to this as you all know what

happens in pipeline if you if you type in the on this you get sort of a mission statement so I went to my house I want to

come in and of how it should look and how it should behave and that is what really this talk is about how to take those principles and apply

them to designing an interface to us to a foreign system traverse a little little motivation what what brought me here and that's

that's a lasting the company that I work for where I actually create a part of a Python client for 6 search search and Alex engine and it's not at

all important for this talk but what is important is we'll be using it for some for some examples of don't be don't be alarmed if you see some trees and somewhere in there so as get to it so important this the design of Python as as I said of guiding set of principles and I've always liked them we know they sound cool and they they

really make sense to me but I always struggle a little bit how do you want actually apply

the have you apply them to code does that actually transfer like it doesn't make any sense In this talk I would like to share how I actually discovered or maybe I just rationalize it away and it doesn't really quite matter probably of how we discovered how we use these principles when I design a new API and how why actually apply

them in real life In because I actually said by

a lot in just in the past sentence a little bit of disclaimer is in place this is of obviously my personal opinion this is what I

find that works for me and also some of you that may have seen my code and documentation or lack thereof so please do as I say and not as I do because I certainly am not enough perfect and uh might code that fully reflects the self and

a could talk begins with the definition of right so API what do we mean by API what we mean by by an

interface so forward for this talk in particular we mean a Python API interface tool of foreign so something that will allow you to talk to a 3rd party system in our case of elastic search and this is something that typically you don't need in Python and Python is complete right you can write absolutely anything in Python so you don't need these interfaces you can communicate directly especially in case of life like Elastic rejection just speaks and she the injuries on you can just use requests or any other favorite HDP library and talk directly but that's not really what you want to do you don't want to keep for discovering the the wheel and that's why we have API it's the sum somebody created an API to

make things simple to but to hide away the complexity so you might talk about an API in this case that it is a

service for for the real goal for the cold that actually does the work for the application code and there is

a huge difference between the the code of the API and the code of the application the real code because the API is really doesn't really apply for a specific use cases because the API doesn't know anything about the real close so it can be used by many different people in many different organizations and hopefully it will be like that's sort of what you hold for when you create an API right that people will use it that it will spread so and the application on the other hand is always written to solve a specific purpose so the API fulfills contract for forward that

code and the contract can be either explicit or

implicit it can either be explicit by documentation you can document the contract can say these are the methods these are the responses and that's probably something that you should have by the way but you can you can also

have sort of an implicit contract of cultural 1 if you please l which sort of of makes the API behave as you would expect that it should be it should be natural and you should be able to rationalize and and to think about the API and avoid any any surprises that sort of what you want for the users of your API to be able to use it as if it were part of the system to not having to think about it too much but 1st of all we need to we we need to address some of the issues with the API the 1st 1 is that the API is bigger

then then application code and yes they actually had to look this up it is an actual word I

wasn't sure for what it is

actually it is actually accepted by the dictionary it the the British protest again what a surprise so it is a lot bigger it doesn't really know what it's going to be used for an application coder-specific it solves and it solves a problem you create an application to scratch an itch to add to that deliver a solution to your customer or to power your product but your API can be used in so many different ways that you never you never know and making any sort of assumptions on how your API is going to be used is can be very can be very difficult so ultimately the API is just a tool and you always have to keep that in mind that it's a tool that should be a but that anyone should be able to

wield to create action and it's a tool to simplify access and to

simplify access the crucial part there's the and we have our 1st line from this apply for simple is better than complex complex is better

than complicated and that is the purpose of the API to actually facilitate this this line to take something that would be complicated opening of socket creating an HTTP headers sending it over grading adjacent body sending it over receiving some responses determining if everything went OK etc. etc. that's definitely unnecessary complex and that applies to

to anything if you want to work with HTTP you also probably wouldn't start with Ross and but you would use something like requests of of of well-designed library that just gives you exactly the functionality that can mean in a way that you can do that you can you so that's that's what the simplifying access is about it's about hiding

complexity so this is a query for 206 search don't worry if you if you don't understand it but essentially what this is is I'm looking for something that at that matches Python and the title it must not match later in the description so I'm looking for release I'm of filtering only those packages in categories search and I wanted do some aggregations I wanna see the distribution of tags and the maximum lines so just some just some query assuming assuming a dataset and that's where is not important what is important is that there is a lot more things on the screen then what I just write and I recall that this does In

the end I could just printed out with the relevancy score and the and the title of the words of the document so there isn't really a lot a a lot of things actually going on but there is a lot of things that are being tied there so how do do we how do we simplify that how do we hide the

complexity well this is what I came up with and I'll spend the rest of my talk explaining to you why and how liking what with that I always see some people cringing because they fuse the school before it's never a good sign OK so In this case I try to extract only the things that are relevant only and the stuff that I actually write only the action items are not really not really all that great all the boilerplate code and to me that's what an API should do it should hide away all the boilerplate while leaving all and that's that's the crucial part living all of the important parts they're just some but all of the important parts and and that sort of the

embodiment of another line of this and thought the explicit explicit is

better than implicit so while I I did away a lot of the complexity

i didn't hide away the crucial parts and that's sort of the very important decision what to hide and what not to 5 because you can always go

then the next step you can always imagine how you could make this even simpler than for example just creating some sort of query language where you just say the 3 words and say this should be there the student and this must be there as a filter etc. but at that point it's it's getting hard to read it's getting harder to get into it's getting harder to reason about so be explicit and and in in the word of cold what that means is when you're hiding from of complexity to hide the mechanics do not hide the meaning so if you're doing an HTTP request with request you still have to know what is the difference between GET and POST and people that is the meaning that is so the mechanic which you don't really want to know about this about sockets and and passing of HTTP headers you just want a convenient access to it so that's mechanics that something that you should hide away that something that it's not specific to the problem that specific to the implementation so this is sort of how will help you to uh how you draw the line looks to high in 1 of 2 so if

we look into into the the origin along the orginal code and so you get you see I've highlighted the parts that are actually just the mechanics I'm sorry about this and where you see that I would I would call query which which is we branches must must not and filter and those can be those can be very confusing and also they don't matter they don't come they don't carry any information they're just out of the way that the query DSL the language and search users based on trees and how it expresses how to combine other query it is it is the

house is reasonable walked in the case this is not what you want to do this is how you want to do it and this is something that I I don't want my of my users to have to know In order to use my life however I am fully comfortable and forcing them to know all the rats so ieee that's all

that I had I never I never had the match or for the term wearing so this is the

meaning you still have to know what is the difference between a term query match query and how to do how communication in this case the negation is in Python so we should all be familiar with it but that's the meaning that's the same as requests still asking you to know the difference between age you get an HTTP POST if I if I had this way it would make

a lot of a lot of people's lives easier but then you would have a very narrow ceiling after which there's nothing there's nothing you can do and also this means that I don't have to teach my users everything they can just use the skills that they already have by understanding the query itself secured itself on the other side the request example of people understanding each GPU nation methods I don't have to reinvent that it's already there

people already know it so all I have to do is give them access to In a similar thing goes for forward goes for the results

here I have some some crazy dictionaries with with underscore source and underscore score that can be a little difficult and again it just there's no meaning 90 per cent of the time just 1 access to the fields or alternatively the matter fields so again just abstract away hided and also provide a more convenient way to access so instead of square brackets and quotes and underscore source in every single line just use the book title so this is something that just simplifies the mechanic while not actually taking anything away also and it's good to fully admit to the link so In this case still showing all the different all the different query types of all the different aggregations you see that I still force of the users to name their obligations just as they would in the in the word cell so

they can them and get them back in the results it's a very thin extract layer exactly because I want people who use search in some other context maybe from another language may be directly through the browser or the command line I want them to be more effective I don't want them to have to learn to get another told to do what they already can do and also once they learn this true I don't want that to go to waste if they have to change something

else or if they have to ask for advice line if I if I created my own complete where language and then I had to ask someone on the internet so this is what I'm doing that all 6 search can you help me all the answer will be no because nobody else will understand that world so the standardization the aliki extraction here is very very important and very deliberate and it is also because well I'm lazy and I don't want to rewrite the entire documentation of all 6 search what word of what was that so that's sort of

a another guiding guiding principles be familiar present to the user something that they that they know from somewhere else whether it said that whether it's a universal concepts like of different types of

HTTP requests or a different types of queries gnostic search or something that's already been used before so don't be afraid to just shamelessly from that you've seen around so essentially could

some of the the library that created the security itself as a combination of these 2 things and the 2nd 1 we've already seen that just using the the role of search API and the 1st 1 well that's jingle words so that they borrowed some some

patterns from GenGO some patterns from all 6 search and combine them together so for example that the training is the fact that every additional filter whenever you add a filter you will get return a copy of the of the query that comes from that comes from that's something that people expect or at least are a familiar with and then you have all the different all the different query

types that come from from our success so hi from people who want to use 6 search should be familiar with both of these concepts and the API should feel home to them should feel familiar they shouldn't be surprised and they should be they should be effective so

one-sided that turn into and to

another will be consistent once you figure out your approach to stick to it so special cases on special enough to break the rule so every single method that you have on the list of all the

search object in our security so will return you copy the the training works of justice expected n the other important part that people sometimes forget about when

talking about consistency is the naming main things consistently linking consistently with other systems much more exactly within your system always call it the same and that is both in your code and also in your in your documentation and of course only do this if this makes sense because practicality always beats security in in Python like we are we're practical people we you were not really interested in the in the your of purity

just for just for its say so don't be afraid to make an exception like trying to make 1 but if you have to that's OK so in our case we have we have the queries on some of the top that follow the pattern every time every call to the method will create a clone of the query object of mutated and return it however when I try to do this for for

obligations this just didn't work because aggregations can be nested and the 1st way how people of me including tried to represent nested aggregations would be just in this nested training called and at that point all broke down I couldn't no longer be creating copy after every after every call so I had to I

had to break the pattern so when you access as good and this is actually done in place so don't be afraid to break the world try not to but also keep in mind that it might happen that you just will have to and that's OK there's a Python cells so and a lot of people than me wrote that so I'm OK with that

another very important rules when designing an API is be friendly and friendly to your users and on both sides the friendly to the users of your API but also be be friendly to the system that you're trying to to simplify access and 2 means are the only known only as part of partly obvious because it's a pretty obvious to realize that Python is interactive a lot of people who use

Python from stuff like I like or the use

fancy used to explore the API eyes and you should you should be able to support its you should be able to hold them with that of by providing them all the tools that they could not that they could ever In the case of

Python those those are the 3 main 1 of their representation and then docstring which if if implemented properly will greatly held with no with actually allowing the users to explore that uh the API and start using it both the beginner users who just into your code for the 1st time in our just exploring around

and also the advanced users where this can greatly speed up the speed up the process of of for example that the representation of string

is often underestimated and can be super used of 1 of the most common questions together with elasticity of cell I had this crazy query in injuries on that somebody wrote or some

other generated how do I express this mean non-security itself policy that's easy you just create a query out of it by wrapping it in the in the few objects and just

ask for the representation and what you get back is exactly the code that will be that it would take to reconstruct that

just using just using the DSL library so that's what that really the representation strength should be

of a representation of the object that you can essentially retype into Python and get the same thing In some cases it's of practical if you have large objects obviously or you have something that can only exist at this once it's not that useful but in this case it is definitely definitely a very useful and it saves me a lot of time because this is something that I myself use quite often that I have this I have this crazy dictionary containing core user query with 10 different sub-queries and 50 obligations and I want to manipulate it and manipulating the dictionary itself is is quite difficult that's why fiber in the 1st place so I just rapid in the rapid and then you just get the query object that can work with and

then when I want to know put it in my code I can use the representation and and put it in there and of course of course docstrings b be nice to your users provide even some examples in your groups think those are the most useful if you if you're reading if you're reading

of header over method like this it's pretty well pretty evident what it does but if it were anything more more involved with more parameters is always nice to include an example right there both for when somebody is reading the code and also when somebody's just looking in their IPython on what's what's actually what's actually going on so that's 1 part by funding interactive that's the more obvious what the 2nd

part is enable iterative built because again there's an applied on teaches us that flat is better than nested and sparse is better than that so does that mean if you if you build something up if you if you

wanna build a sophisticated querying to also search you keep adding clauses 1st filter on this and on that and this aggregation in and if the user requested this filtering and this kind of filter so there is a complicated state that you need to remember what the query currently looks like it's a sign of a polite to PPI as I would call it that it doesn't force you to remember the state but it can store the state internal so you can use the API from the get go and start building your theory in this case for your plants in other cases and so you don't have to store everything yourself so in our case it looks like that's you can also

see that this enables for nice practices like commenting the code and actually explaining to you to the users what they're doing and I can go on line by line and very easily very easily be constructed even if you're new to the new to us search you would probably be more or less able to tell what this does especially with the problem so sure they're still some there so some magic there's still something that specific prospects we've talked

about the term and match in all the different weirdness of this and that's what overall this should not be surprising I I want to only filter category search and I want to match the title by following etc. etc.

but the most important part is I don't have to 1st build up some weird dictionary containing all the keyword arguments or represent the state in any way I can just keep creating and the adding to the search object can be be happy with that so iterative build its of this often something that's that's

underappreciated because it allows you to not care about this about the state which can be which can be very hard and finally when we're talking about being friendly safety is also very friendly you should always fatal explicitly unless unless explicitly silenced so all your

defaults I should I should actually be the safest possible so if you have

if you have any option failed for example 6 search will always give you a response even if only 20 per cent of

your data is available it will give you response and say OK I only see 20 per cent of your data but here have some have some results anyway and then it's up to the user to decide whether that's good for them or that the when faced with the decision like this what do I do I fail in this case or is it OK and going to leave it to the user I saved always fail but allow the user to override allow the user to say I am aware that this is a situation that might happen and I don't care but at that point you force the user to explicitly the

responsibility to maybe even do some some research you notice that that's something that of that I've repeated quite often during in this talk to not be afraid to force the user regular something about what they're doing and this 1 it's article important because take it deals with the just with safety and it helps prevent nasty surprises once the user moves to production also think about how

you test your code and how other people the code provide some

sort of of dumbing interface or maybe just as a set of test in this case classes of 4 for the your favorite testing libraries out there so that other people have a lower barrier to entry to do some testing essentially a beta testing don't made the testing as simple as possible because that's what you want to do ideally you already have some code like that's somewhere somewhere in there already because you should be testing the API so it's only a matter of expose the exposing that hot also to to your user journal test helpless or something In the final

chapter is about API still being cold it's different from an application code and we've we've highlighted the reasons house but it's also similar in in a lot of cases it needs to be tested things can change and you might need to you might need to adapt also there is a there is a lot of a lot of decisions going into how do you decide what goes

on and what doesn't what are what are the features that you want as part of your library as part of your API and what you leave serve as an exercise to the user do I provide this set of helpers y of y expose this functionality this this parameter or is it only used by by a very few people and there there are several several ways how to do that 1st of all is the actual decision but 2nd is that my favorite to avoid that decision and allowed the more advanced users to sort of always step away

a little bit and provide them access to the lower leg lower layer the more low-level API

In case of ElasticSearch I can always go 1 step back and just sending in of all dictionary if I don't want to create the the query using the iterative syntax that I just showed you I can always just created myself and at that point I don't have to care what options are

supported by the external and which are not going just do everything manually and send it back the same again it's not a novel idea it's the same with the doctor method on jingle various that's where you can just send in a sequel query when you don't want to rely only on to generate 140 so always allow access to the lower level if at at all possible and also at MIT that no code is

perfect and that goes especially for uh especially for API eyes because there are then than the actual application and when deciding what to include and what not but keep in mind the last line of survival that will deal with various now

is better than never will never is often better than right now and that's the that's the important and that's an important thing but when you don't know whether to include something or not it's perfectly fine to say no especially if you if you have a way for the user to move around you and so I always prefer to give them away or I'll Michael then to try and support all the different possible avenues through Michael because that can lead to a nightmare that can lead to a nightmare for my users with an overwhelming amount of options and Beverly foreign aid In a nightmare for me supporting all the different all the different combinations so so think about what makes sense and how hard it is to do it without

direct support in your if it means that when I'm not supporting this option the user will have to create a dictionary and send manually I'm OK with that if it means that they'll have to instantiate a new connection and talk to something directly and and do some other complicated stuff maybe not that much and I want to I want to provide them the functionality directly so have that's sort of the last part of how you decide what to support and what not to support and now I believe we have 10 minutes for questions and if you don't

get your questions answered or you just want yell at there is money there's my Twitter and if anybody

has any questions I think you were

at all and so the question so if we create a great API with expressive that expressiveness impacts don't you think that we're making the internal sources of the unit API more complex than we are to maintain can we enable itself and I guess there

there is there is no not necessarily a correlation between the complex and and the and the ease of use yes sometimes you might need to resolve to more complex things like using metaclasses and descriptors if you want to make things nice but it's not it's not necessarily true so it might happen but it's a but it doesn't have to there is there is no direct correlation thanks anyone else 2 gentlemen and

and what role do you think emulating built-in objects all using like Python types by the Boeing types comes into the uh writing a Python if you for

example when like when should you do you think that you should use dictionaries not supposed to cheese instead like material DSL for your particular thing as see so I I prefer to of again decide what can you take away and then compare the results so if if I use raw dictionaries and if I if I create my own thing what part of the pain goes away is enough to justify and the extra dependency the work that will go into the API etc. just sort of do this exercise in your mind like OK these things I can I can extract the way these things I can generate automatically so the user will only have to put in these these 4 things and then compare the results it might be that the differences wouldn't be big enough to justify creating a DSL at creating of of library from of both the point of putting the working and also forcing the people to to learn that so that's that's what I do like every every time i've sort of mock up how the API would look like and then I compare is it worth it or is it not to make sense the degree of influence of

questions in the beginning with questions you can just grab the confluence even begin to impose from here