We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Leveraging documentation power for better web APIs

00:00

Formale Metadaten

Titel
Leveraging documentation power for better web APIs
Serientitel
Teil
73
Anzahl der Teile
169
Autor
Lizenz
CC-Namensnennung - keine kommerzielle Nutzung - Weitergabe unter gleichen Bedingungen 3.0 Unported:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen und nicht-kommerziellen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen und das Werk bzw. diesen Inhalt auch in veränderter Form nur unter den Bedingungen dieser Lizenz weitergeben
Identifikatoren
Herausgeber
Erscheinungsjahr
Sprache

Inhaltliche Metadaten

Fachgebiet
Genre
Abstract
Rudy Sicard - Leveraging documentation power for better web APIs Web APIs that are easier to understand, develop, test and use, is a popular subject. "An API is only as good as its documentation". We decided to play with this proverb and leverage the power of documentation. We propose to use the code documentation and the type system to provide lots of free features: explorable APIs, better error messages, automatic testing. Python is perfect to explore code and documentation dynamically. We'll demonstrate what we came up to and the lessons we've learned. ----- 'Rich' web APIs that are easier to understand, develop, test and use, is a popular subject. There are a lot of new specification languages (e.g. swagger, apiblueprint ...) and libraries (django-rest-framework, drf ...) that provide features in this direction. Following the old proverb "An API is only as good as its documentation", we decided to play with these ideas and focus on leveraging the power of documentation. We propose to use the code documentation and the type system to provide: - browsable APIs, that are easy to interact with and visualize, reducing the need to provide custom UIs - verification of inputs/outputs along with precise error message if needed - automatic [de]-serialization of inputs/ouputs outside of the domain code - smart exception handling, e.g. exceptions that are not documented are automatically converted into internal errors - automatic testing, e.g. input, output and result including exceptions are tested ensuring the code works and the documentation is up-to-date. This use case is one of the rare situation where introspection is desirable and unavoidable. And Python is a good language to explore/exploit code and documentation dynamically. The perfect excuse to spend some time on meta coding a first implementation while being at work. We'll demonstrate what we came up to, the advantages and limitations compared to other approaches. And we'll share the lessons we learned from this experiment.
11
52
79
Dämon <Informatik>W3C-StandardLeistung <Physik>Kontextbezogenes SystemLeistung <Physik>VersionsverwaltungBetrag <Mathematik>Zentrische StreckungHardwareSoftwareComputervirusVorlesung/KonferenzComputeranimation
Kontextbezogenes SystemCoxeter-GruppeDifferenteSoftwareWeb ServicesBrowser
BrowserFramework <Informatik>Design by ContractInterface <Schaltung>UmwandlungsenthalpieDienst <Informatik>Design by ContractExploitParametersystemPunktMomentenproblemComputeranimation
Ein-AusgabeNichtlinearer OperatorTVD-VerfahrenValiditätInstantiierungDienst <Informatik>Vorlesung/KonferenzBesprechung/Interview
Chi-Quadrat-VerteilungKreisringMetropolitan area networkBildschirmmaskeEin-AusgabeNichtlinearer OperatorPhasenumwandlungDifferenzenrechnungExogene VariableFamilie <Mathematik>ParametersystemDifferenteSystemaufrufResultanteSichtenkonzeptXML
Chi-Quadrat-VerteilungKonvexe HülleFunktion <Mathematik>Umsetzung <Informatik>ParametersystemBildschirmmaskeDreiEin-AusgabeCodeFunktion <Mathematik>ValiditätCASE <Informatik>Mailing-ListeUmsetzung <Informatik>Nichtlinearer OperatorTypentheorieFokalpunktZeichenketteRechter WinkelXML
MultiplikationsoperatorTabelleEin-AusgabeClientBildschirmmaskeValiditätDateiformatBesprechung/Interview
Statistischer TestVerträglichkeit <Mathematik>Offene MengeDesign by ContractAdditionRegulärer GraphImplementierungDienst <Informatik>Statistischer TestUmwandlungsenthalpieFormale GrammatikComputeranimation
BildschirmmaskeStatistischer TestVerträglichkeit <Mathematik>Umsetzung <Informatik>Innerer PunktSyntaktische AnalyseEin-AusgabeAusnahmebehandlungFunktion <Mathematik>Automatische HandlungsplanungFunktionalEin-AusgabeTypentheorieAusnahmebehandlungInformationPhysikalischer EffektMapping <Computergraphik>QuaderMultiplikationsoperatorDefaultAggregatzustandElektronische UnterschriftFehlermeldungPunktWiderspruchsfreiheitBildschirmfensterMehrrechnersystemNichtlinearer OperatorVorlesung/KonferenzComputeranimation
Innerer PunktSyntaktische AnalyseUmsetzung <Informatik>AusnahmebehandlungFunktion <Mathematik>Ein-AusgabeTypentheorieNebenbedingungDatentypMailing-ListeBoolesche AlgebraZeichenketteGanze ZahlZahlenbereichSchwimmkörperObjekt <Kategorie>TypentheorieGrenzschichtablösungParametersystemArithmetisches MittelInnerer PunktTeilmengeDatenstrukturValiditätObjekt <Kategorie>Umsetzung <Informatik>Ein-AusgabeLeistungsbewertungDienst <Informatik>GamecontrollerRechter WinkelNebenbedingungDateiformatVorlesung/KonferenzComputeranimation
Boolesche AlgebraZeichenketteGanze ZahlInnerer PunktZahlenbereichSchwimmkörperObjekt <Kategorie>Umsetzung <Informatik>Funktion <Mathematik>DatentypNebenbedingungAusnahmebehandlungCodeImplementierungParametersystemGewicht <Ausgleichsrechnung>Ein-AusgabeDesign by ContractKlasse <Mathematik>DatenstrukturProgrammbibliothekValiditätUmsetzung <Informatik>AdditionFunktionalCASE <Informatik>TypentheorieNebenbedingungData DictionaryDienst <Informatik>AusnahmebehandlungGruppenoperationTemperaturstrahlungRationale ZahlMagnetbandlaufwerkBildschirmmaskeWiderspruchsfreiheitVorlesung/KonferenzComputeranimation
ResultanteFunktionalEreignishorizontDesign by ContractInhalt <Mathematik>AusnahmebehandlungSystemaufrufDienst <Informatik>TypentheorieVorlesung/Konferenz
BildschirmmaskeParametersystemSkriptspracheAppletTexteditorTemplateWeitverkehrsnetzKrümmungsmaßTexteditorSummierbarkeitAbstandCASE <Informatik>Nichtlinearer OperatorInhalt <Mathematik>DefaultBildgebendes VerfahrenTypentheorieSerielle SchnittstelleDeklarative ProgrammierspracheÄhnlichkeitsgeometrieSystemaufrufSichtenkonzeptTemplateFlächeninhaltBildschirmmaskeValiditätComputeranimation
AusnahmebehandlungAdditionInstantiierungSoftwaretestSystemaufrufStrategisches SpielGeradeKategorie <Mathematik>Umsetzung <Informatik>Ein-AusgabeFunktion <Mathematik>DatentypElektronische PublikationStatistische HypotheseBildschirmmaskeFlächeninhaltSoftwareentwicklerStatistische HypotheseMessage-PassingStrategisches SpielDesign by ContractGenerator <Informatik>Dienst <Informatik>Nichtlinearer OperatorRandomisierungAusnahmebehandlungInstantiierungCodeSoftwaretestTypentheorieEin-AusgabeProgrammiergerätTypprüfungStatistischer TestZeichenketteRelativitätstheorieWort <Informatik>MathematikCASE <Informatik>Inhalt <Mathematik>AbstandVorlesung/KonferenzComputeranimation
Mailing-ListeZeichenketteFunktionalVorlesung/KonferenzBesprechung/Interview
ImplementierungComputersicherheitWasserdampftafel
AusnahmebehandlungAdditionGeradeElektronische PublikationStatistische HypotheseStrategisches SpielInstantiierungSoftwaretestSystemaufrufKategorie <Mathematik>Umsetzung <Informatik>Funktion <Mathematik>DatentypEin-AusgabeWeb SiteExogene VariableVerträglichkeit <Mathematik>SichtenkonzeptVerschlingungFramework <Informatik>ZählenZellulares neuronales NetzBildschirmmaskeGraphfärbungFramework <Informatik>Virtuelle MaschineMAPExogene Variablesinc-FunktionDisjunktion <Logik>XML
MereologieExogene VariableSystemprogrammSchlussregelSichtenkonzeptInstantiierungTouchscreenVisualisierungVorlesung/KonferenzBesprechung/Interview
Verträglichkeit <Mathematik>SichtenkonzeptEin-AusgabeVerschlingungFramework <Informatik>Exogene VariableZellulares neuronales NetzMathematikZeitstempelGanze ZahlSichtenkonzeptFahne <Mathematik>Formale SemantikExogene VariableEntscheidungstheorieFunktionalPhysikalisches SystemComputeranimation
Verträglichkeit <Mathematik>SichtenkonzeptVerschlingungEin-AusgabeFramework <Informatik>Exogene VariableZählenDifferenzengleichungMAPParametersystemChi-Quadrat-VerteilungBildschirmmaskeVerschlingungSichtenkonzeptEin-AusgabeInformationExogene VariableNichtlinearer OperatorDateiformatFormation <Mathematik>MathematikStabDruckverlaufVorlesung/KonferenzBesprechung/InterviewFlussdiagramm
BildschirmmaskeNichtlinearer OperatorVerschlingungWeb-SeiteDatenfeldVorlesung/Konferenz
Offene MengeTypentheorieStatistische HypotheseCodegenerierungProgrammbibliothekClientSummierbarkeitTaskUmsetzung <Informatik>Statistischer TestBildschirmmaskeCodeImplementierungClientMessage-PassingDienst <Informatik>StandardabweichungOpen SourceStrategisches SpielStatistische HypotheseDokumentenserverFehlermeldungSoftwaretestTypentheoriePaarvergleichRuhmasseGenerator <Informatik>Design by ContractProgrammbibliothekMetropolitan area networkAbstandAdditionGemeinsamer SpeicherGruppenoperationInhalt <Mathematik>Coxeter-GruppeQuellcodeComputeranimation
CodeWort <Informatik>
Versionsverwaltung
Transkript: Englisch(automatisch erzeugt)
Welcome, we have Rudy here, showing us how to leverage documentation power for better APIs. Give him some clap.
Hello. So, hello, I'm Rudy, I'm working at Criteo, a company that delivers personalized ads
at scale. My team is building monitoring tools for our large software and hardware inventory. And sometimes, we decided to make monitoring as a service and start to experiment different
ideas around web service and browser APIs. So this presentation gives a glimpse of what we have done. And this is a joint work with my colleague, which is there.
So, what is a browser-able API? Obviously, it's composed of two worlds. The browser-able world, so, probably it's related to a browser, and, more importantly,
it's related to exploration and experimentation, and I will add to that more later. And there is a world API, we've got a formal talk, so an API is a kind of contract, it
can be very explicit, like if you're declaring a swagger specification file, or it can be implicit, like in the documentation. The contract defines the valid use of the service that is linked to the API, or the
operation you can do with this service, and also what are the valid inputs for the operation in the service. So why is it interesting? Because it brings a lot of features for your service.
So the first feature I will speak about is why we rely on the documentation to describe the API. Then I will speak about a basic feature that you have in browser-able API.
So the first basic feature is the exploreability, it means that you can go to the API and ask what the operations are possible in this API. So here you have a view of an API with different operations, and then you can just
click on an operation, and you see the documentation of that operation, the parameter types, and what are the possible results. And then, more than that, you can play with an operation, so you have to see the input of the operation, and call it, and so you can see what is the response and
explore the behavior of this operation. But still, typing the input, like a JSON input, can be difficult, so we probably can do something
better. Another really important feature of an API is to enforce the validation of the input on the input and the outputs. By the way, this talk focuses on JSON inputs.
You don't want an operation to be used in an adapted way, that's why you should validate all the inputs of your service, like if a list is transmitted instead of a string, nobody knows how the code will behave in that case. And even for valid JSON inputs, you may want to have a conversion between the JSON inputs
and between your Python types. Okay, this one is an obvious feature, like I said before, it can be really painful to
enter JSON data, because it's a verbose format. So, having forms makes you save time, and it brings client-side input validation on the table.
Another really interesting feature is that you have that kind of API contract, and you have your service implementation, and you want to be sure that they perfectly match, so that would be really cool to just ask for automatic tests for checking that they match.
And I will also present another feature that makes browser-able APIs more user-friendly, and last, just to mention it, you may want to generate, even if you use the doc, more formal specification contract, like in Swagger, because Swagger has a lot of amazing features
that you can later use, so, and this is the plan of the talk, in fact, so I'm starting now. So, imagine you want to add a new function to your API, what do you do?
Okay, it's easy. You document the functions, and you add the publish decorator on it, so this decorator will register the function as an operation in the API. The function will be left unmodified, so you can still use it as before, when the
decorator was not there. And behind the scenes, the function will be wrapped in the API, where all the data about HTTP will be handled.
So how is it done? Documentation will be parsed, so that we have information about input, output, and exceptions. This gives you type concerns about inputs, outputs, and possible zero, and also the
signature of the functions give you default value for inputs. So here we have type concerns, that's what's in the red boxes.
Type concerns, they are more than just type names. They can be parametrized for iterable and mapping types, like here we have a parametrized list, that is parametrized by mytb, or it can also contain several type names, like
the first parameter, it's either an int or a swing. And from that type concerns, we can find all the input should be validated and translated
from JSON data structure to Python object. For that, we convert that type concern to a JSON schema. So for those that, for people that don't know what is JSON and JSON formats, a JSON object
can be seen as a subset of the built-in Python types. So often the conversion between the two is free. But for your services, you still need to do, you may need to do additional validation. And this is done by a JSON schema.
So what is JSON schema? A JSON schema, it describes the structure of the expected JSON object, like its types or its minimum value, what it contains, like the name of the field, the one that
are optional, so it really describes what you are expecting as an input. And there are also situations where you want real conversion to happen.
So you have, like, a JSON dictionary and you want it to be converted to your own Python class. So for these two cases, when you want to have additional validation and custom conversion,
we introduce two classes that helps to attach the conversion and the JSON schema to your regular class. So from that, we are able to retrieve a JSON schema from any type constraint.
So what does the validation and the conversion happen? When the service is called and the inputs are received, there's first a validation. For most basic validation, we rely on the type constraint in the documentation.
And for more complex ones, we rely on this JSON schema. So there's a library called JSON schema that you can use to validate any of the
data structure with your JSON schema. After that validation, there is the conversion step. That is either implicit for most basic type or uncrafted using the from JSON cases.
And then the real function is called. And from that, there can be two outcomes. You can have an exception. And so what we do is that we check that the exception is respecting the contract. If it's not, the content of the exception is hidden because we don't want the user of the
service to see anything that is not accepted by the contract. And at last, if we get a result from that function call, we validate it based on its type.
So what else can we do with JSON schema? After all, it's a sundown. So you can use JSON editor. It's able to use a JSON schema to generate a form, which is also an editor.
It's able to validate the form and give you back the validate JSON value. So you just have to plug in a call to JSON editor in your HTML template.
And you have a cool feature for almost nothing. So here you have a view of what you had before. You had to type the JSON content in the play with me text area. And now using JSON editor, you just have a form.
So it's really easy to create an input. And note that there are similar features in Django REST framework, but you should also, in this case, declare things manually, like the serializer for the form.
Just like you have to write a JSON schema here. So what else can we do with JSON schema? I don't know if you know a package named hypothesis. It's a package that helps you to do property-based testing.
So it's composed of strategies that are able to generate random inputs for many usual types. And it's really a fantastic tool.
I encourage you to use it. So let's add the generation strategy for JSON schema. With that, we can generate tests to verify that all the operations of our service respect the contract described by the documentation.
So here, I've put two examples. So I've taken a valid code that respect the contract of the API, and I've modified it. So in the first case, I changed the code just before it was raising an exception.
I added a type error, like I had an internal string. And from the automatic test, we have this message where, first, the type error is discovered
and is shown to the developer with a proper stack trace. And second, the programmer is also warned that one of the exceptions that is described in the contract is never occur with this operation.
So it shows that what is described in the documentation could be outdated, for instance. And then I've put another example where I modified the valid code and make it return
a bad value, like a list of string for a function that is expected to return something else. And so what we see is that it's also detected automatically.
So we have something that is able to take your whole API and automatically check that what you have described in the documentation is on par with the current implementation, which is a really cool feature.
So what can I do next with JSON schema? I don't know. We stop it there. So we already have an easy way to enter JSON with forms. Can we have something useful to visualize the JSON response?
So taking inspiration from Django RAS framework that already does the coloring of the JSON response, we decided to go further and to add something that we called pretty view.
So a pretty view, it helps you to define custom visualization of the JSON response. So here, for instance, so you have the overview above and the pretty view in the lower part of the screen. And you can see that the JSON contains a timestamp.
And in the pretty view, the timestamp is converted directly to a human readable date. And there's also another example. You have an integer value in the JSON, and it has a particular semantic.
And the pretty view is able to show that semantic. So here it's the value zero, which is an OK flag. So it really makes the JSON response easier to read for humans.
And another thing that we added to the pretty view is the ability to add extra information.
Like here, the pretty view says something about active downtime that are not in the initial JSON response. And moreover, it adds a link near that extra information. And that link, which we called an input link, it's a link that is able to go to
another operation and pre-fill all the information in the form. So you can navigate between different features in the API. And you don't have to tip anything. So here, when you click on the red link, you will go to the operation page,
to another operation page, and every field will be already pre-filled. So what's next? We plan to open source all the things that we presented there.
So you can watch the Criteo GitHub repository. We will add the different contribution there.
We will also contribute to hypothesis with the JSON schema strategy. One thing that we didn't do, and could be very interesting, is to support NN natively, because it's a really interesting type of place.
And to integrate more with PyTests, the thing that I showed to you that gives you like a diagnostic for your API, like with the warning and the error about mismatch between the contract and the implementation.
Currently, you have to run it by hand. And it would be better if we integrated the standard testing package. Then there is the Swagger spec generation. So we already have a lot of cool features.
So why would we want that? Because Swagger has the ability to generate a client code library. That means you code your service in Python, and then you want people to have automatic access in Java. You just write the Swagger file,
and it will generate the client for you in Java. So that's really a good thing. So if you have to only keep a simple message from this presentation,
possible APIs, they are really interesting stuff, because they are very user-friendly, and you can discover the content and explore and test, and it's very good. The documentation and the types,
they can be really used to add a lot of features to your services. So here I give some examples with the features I present, but there are a lot more things to do.
And also the conclusion, Gson and Gson schema, they are very handy, they are standard, there are a lot of tools around them. So don't hesitate to use them. I think it will help you to easily have new features
in your code. And that's all, if you have questions. Questions?
Thank you for coming.