ScanAPI
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 130 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/49942 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 202061 / 130
2
4
7
8
13
16
21
23
25
26
27
30
33
36
39
46
50
53
54
56
60
61
62
65
68
73
82
85
86
95
100
101
102
106
108
109
113
118
119
120
125
00:00
Different (Kate Ryan album)Lebesgue integrationWeb-DesignerProgramming languageField (computer science)NeuroinformatikSoftware testingFront and back endsLogistic distributionChainComputer scienceFocus (optics)Information securityE-learningXMLUMLComputer animationMeeting/Interview
00:53
Lebesgue integrationTrailRow (database)Point (geometry)Software testingInheritance (object-oriented programming)
01:23
Software developerComputerEvent horizonException handlingDisintegrationError messageClient (computing)Field (computer science)Neuroinformatik2 (number)Point (geometry)InformationField (computer science)Software developerProjective planeProduct (business)SoftwareOperator (mathematics)Self-organizationRemote procedure callSoftware bugFile formatTelecommunicationStructural loadMultiplication signExistential quantificationCharge carrierDebuggerFront and back endsError messageIntegrated development environmentService (economics)Logistic distributionChainContext awarenessLebesgue integrationClient (computing)Thermodynamisches SystemDifferent (Kate Ryan album)TrailEvent horizonSoftware testingLength of stayNP-hardXMLComputer animation
06:31
Software frameworkOpen sourceSoftware testingDisintegrationFluid staticsBitSoftware testingUniform resource locatorEmailSocial classImplementationObject (grammar)Internet service providerStatisticsPoint (geometry)CodeDependent and independent variablesTraffic reportingResponse time (technology)Open setCodeTotal S.A.Field (computer science)Term (mathematics)Error messageResultantLatent heatCASE <Informatik>Computer fileLengthPOKELibrary (computing)Electronic mailing listIntegerSoftware frameworkExpert systemMereologyVideoconferencingDifferent (Kate Ryan album)Multiplication sign2 (number)FrequencyProcess (computing)Lebesgue integrationThresholding (image processing)CountingRootKey (cryptography)Web pageLaptopMathematicsPointer (computer programming)Content (media)ChainRadical (chemistry)Open sourceInformationSerializabilityWeb browserPort scannerInstallation artMessage passingScripting languageRouter (computing)Computer animationSource codeXML
14:42
Fluid staticsMathematicsTotal S.A.Digital filterDependent and independent variablesVariable (mathematics)Thermodynamisches SystemInformationComputer fileTemplate (C++)Patch (Unix)Visualization (computer graphics)Group actionLattice (order)Software testingVisualization (computer graphics)Formal languageKey (cryptography)Lebesgue integrationDependent and independent variablesInformationIntegrated development environmentJava appletVariable (mathematics)Computer fileMedical imagingConfiguration spaceMultiplicationTemplate (C++)Common Language InfrastructureLatent heatLevel (video gaming)Software frameworkProjective planeProcess (computing)Continuous integrationCodeWorkstation <Musikinstrument>PlastikkarteStructural loadCircleOrder (biology)Element (mathematics)Different (Kate Ryan album)Traffic reportingResultantOnline helpGoodness of fitImplementationMathematicsDatabaseProduct (business)Software testingContext awarenessSoftware maintenanceLibrary (computing)Front and back endsMultiplication signPort scannerOpen setRaw image formatGroup actionWebsiteRevision controlLine (geometry)Self-organizationArithmetic progressionType theoryLogistic distributionTwitterCodeLink (knot theory)Sinc functionConnectivity (graph theory)Spring (hydrology)Computer animation
22:36
Touch typingProjective planeTwitterComputer animation
23:02
TouchscreenZoom lensMultiplication signMeeting/Interview
23:23
Key (cryptography)Integrated development environmentLebesgue integrationPresentation of a groupTrailOpen setWordVariable (mathematics)AuthenticationLine (geometry)InformationDataflowStructural loadProjective planeConfiguration spaceOnline chatPort scannerRaw image formatPlastikkarteMeeting/Interview
Transcript: English(auto-generated)
00:07
So, coming up next, we have Camilla Maya. She has been coding professionally since 2012, mostly with the Ruby and Python programming languages in the web development.
00:22
Presently, she's developing mainly with Python, has passed through different business fields during her career, like computer science, or computing security, e-commerce, e-learning, and finally and currently, logistics and supply chain. Backend is her primary focus, so she's going to be talking to us about automated integration testing and live documentation for your API documentation.
00:46
We all hate writing docs, so I'm sure this will be very helpful to know. So, yeah, let's get started. First of all, I'm super glad to be here. You know, talking in the same track and the same day as you is like such an honor, it's like insane.
01:04
So, I'm super glad to be here. My first year of Python, first as a speaker, first as an attendee, first of all. So, yeah, and today we are going to talk about scan API and automated integration testing and live documentation for API. Exactly, if you hate to write docs, this is the point.
01:24
So, the motivation. So, everything started in a week, far less one year ago, where I was working as a firefighter in my company. Firefighter in our company means that we are the people that in the team desire to handle all bugs, all fires, all stuff.
01:44
Then the other people from the team of the team can focus on the project. So, I was there only like facing bugs during this whole week. And then, okay, but then you're going to ask me about fixing bugs, what are you talking about? Okay, so let's give some context.
02:01
I work at Loadsmart, it's a logistics company that tries to make together companies that have loads to be delivered with carriers that have trucks and all the equipment to deliver this load. Everything in the United States, so the company, it's an American company, and I work remotely for it.
02:25
Okay, and you already know, I was working as firefighter at Loadsmart remotely, but who am I, yeah? So, I am Brazilian, back-end developer, I have a bachelor of computer information system, I'm coding since 2010.
02:41
I have the most of my experiences with Python Ruby, I have three years of experience with Python. It's a baby in Python community. I helped to organize events, so I helped to organize the first Pyjamas and then the organization of the second one.
03:01
And I'm an organizer of Heropython here too, so I hope you are enjoying the conference. Okay, so you already know me, you already know my company, and I was there working as firefighter. And the thing that I noticed that all the fires that I faced that week, they were like, they have something in common.
03:23
We have three kinds of different fires. So, the first one was integration errors. So, we have some clients, services that we consume for tracking, so tracking the trucks and all the stuff. And we were facing some problems like in this integration.
03:42
So, we were expecting there to send some field and this field were missing or the field was with misinformation. And this communication was not super clear. Also, we were facing some problems with front-end, front-end receiving data from back-end that was not expected.
04:00
Was in a different format, was some missing information there. Okay, so this was the first kind of problem. The second kind of problem was outdated documentation. Because you could say that, okay, but you don't have like a documentation, why didn't follow this documentation? Documentation when it's like manually, it's hard to maintain.
04:20
So, a lot of time we have missing any points or missing fields and endpoints or misinformation, deprecated endpoints that are there but shouldn't, fields that shouldn't be there. So, this was the second kind of problem. And the third one was the fact that it was really hard to recreate the scenarios.
04:43
So, let's suppose I was there as a firefighter and someone from operations tells me, oh, this endpoint here is not working for production. I cannot like try to recreate the scenario in production. I cannot get that same scenario and test it myself because production. I have to recreate it in the development environment or something that is safe.
05:04
And then I can debug it better. So, I try to think here, like let's imagine that we have a company, a flight company, and then we have an endpoint that is called, that is used to reserve a seat.
05:20
And then the operational team says, oh, this endpoint to reserve a seat is breaking in production. Something's not wrong. And then the operation and then the tech team will try like to recreate this scenario in another environment. So, then, okay, but if you have a seat idea and have just reserve, probably you need a flight before because you had to call an endpoint to get a flight before.
05:44
Otherwise, you cannot reserve a seat without a flight. And for this flight, you need to have an airplane. Otherwise, we don't have a flight without an airplane. And the same for a passenger. If you want to reserve something, the passenger needs to reserve this. And if you have an airplane, but is this airplane available for this flight or not?
06:04
So, we have a lot of questions, a lot of difficulties to reproduce the same scenario that the operational team tells us that is breaking. So, you can imagine like we have really complex, and this is that scenario at Los Mairi.
06:21
We have a really complex of chain of requests that depends one on another. And we want to recreate scenario exactly in the middle. So, this is super hard. So, bringing all this problem and this fire that we're facing in this week, I started to try to put together and understand how we can like solve this or at least make it a bit better.
06:44
So, from this, we created the scan API. What is the proposal of it? Okay, so it's an open source framework. It's written in Python, and we have two goals with it. The first one is to provide a live documentation. Here, it's live documentation.
07:01
You understand it better, but it's not the same as open API documentation. It has a different approach. They are complementary. And the second besides documentation, we want to provide integration tests. So, for anyone that has mostly known Pythonology can implement integration tests for the endpoints for this API.
07:26
Okay, so how does it work? Let's get started with one example. Let's take an example of the Pokemon API, okay? So, the Pokemon API, you can access it with this URL. And let's check the first endpoint just to make you use it with the response of it.
07:45
So, I'm doing here HTTP. It could be like a curl or whatever, but it's an HTTP to the endpoint slash v2 slash Pokemon. So, if I run this in my browser, let me grab the pointer here. If I run this in my terminal, then we are going to have the headers here.
08:02
And afterwards, we are going to have the response itself. So, if we hit here now, it says that we have 964 Pokemons. The next URL for the pagination will be this one. The previous is null because it's the first page. The results will be like this list of the first one will be a Bulbasaur with this URL to access the details of the Bulbasaur.
08:24
The second one, Ivosaur. The third one, Venosaur, so on. Okay, so this is the API just to make you comfortable with it. So, how does it work? First thing that we have to do is installing. So, Python, you're a red expert on it, pip install scan API.
08:43
Afterwards, we have to create an API specification file. So, API YAML file, it could be also JSON, but here we are going to present with YAML with this specification. So, the first keyword is API and then we are going to start creating our endpoints. The first one is to represent our whole API, so the name is Pokemon, PokeAPI.
09:05
The path here is the base URL, slash v2. Then we are going to have the endpoints. So, the first endpoint will be related with Pokemon. This name is only identifier. Then we are going to have the path, Pokemon. And inside this endpoint, we are going to have in the request.
09:23
So, the request, we have the name list all. That means that we want to list all Pokemons from this API. And then we will use a method, HTTP method get. And the path will be only slash because it's the root. So, we are going to concatenate here and this will be the result.
09:43
Okay, so the result will be v2 slash Pokemon slash that's it. So, okay, how we can run it? So, basically the one thing that we have to do is to call scan API because it's a command line interface. You call scan API and you pass the API specification YAML file.
10:02
Here we load, we make in fact the request and then we write the report, an HTML file. Here is the example of this running process. So, here you are going to have the endpoints, API v2 slash Pokemon.
10:20
The status code of the response is 200. All tests passed because we didn't implement any, so we are good. And here is the test summary that we are going to talk more about briefly. Okay, and if you want to see the details of this endpoint, we can collapse it and then we are going to see the request information.
10:41
So, here is the difference. Open API is more descriptive. So, it says we should have this field, this field is required, this is not. This should be an integer, the length of this field should be this and that. Here is not this approach. The approach here is more like a reporting about the request that was just made.
11:01
So, here you can see like live real-time information about the request. So, the user agent that was used in the routers of the request, we have also here a curl command that you can like just copy and paste in our terminal and then you can reproduce exactly the same way the request that it's going to API just made.
11:22
And here you have the response details, status codes, response time, if it was a redirect or not, headers for the response. And here, for example, the content. So, here if you see, it's exactly the same content that I was showing in the HTTP command before. So, if first Bulbasaur and then we have Vivasaur and so on.
11:42
And here the test. Since we have no test, no information here. Okay, we talked about documentation. Let's try to understand a bit about the integration test. So, let's get the same request here, okay? And then we have the same stuff, but now we are going to implement test.
12:01
I suppose we want to implement the test that checks the status code is 200. So, the only thing that we need to do is use this syntax. Everything inside this syntax is interpreted as Python code. So, here we can get response.statusCode equals 200. It's going to do this assert.
12:23
If you want, for example, to ensure that our response will be fast enough, we can make another assert to get the Lapsot total seconds and then compare it with the time threshold that you want. Like for example, half a second. Also, if you remember the response of the request,
12:44
we have here a count, an x and everything here. So, let's suppose for a serious reason, that we want to ensure that every time the count will be the same. So, we can grab here response, then we turn this post into JSON and we get the count value with the key here
13:05
and then we compare that will be always the same. And then you might be asking yourself, how can I know what this response object can do? This response object can do everything that a response object from the lib request can do.
13:22
So, this object is a request object from the lib request. It's a response object from the lib request. So, if you want to see everything that this object can do, you only have to check the documentation and see the response class, okay?
13:41
Okay, and if you run it again, the same command scan API and the API specification, then we are going to have the tests here passed. So, we implemented three tests, all passed. We have here the summary saying that three passes, no failures, no errors, and the total time for this script to run.
14:01
Okay, and now if you remember, it's still missing one part to be solved. Thinking about the problems that I brought in this beginning of this talk. So, the chaining requests, how are you going to do that? How are you going to solve? So, let's suppose now that we are going to get, that we want to get the details of a specific Pokemon.
14:20
So, in this case, every time that we hit the endpoint slash Pokemon, the list that will be returned will be the same, like first Bulbasaur, second Ivosaur, and so on. But the most of case, this list is not static. This list change, for example, if you try to get the flights during a period of time, I don't know, the flights between tomorrow
14:41
and the next weekend, you will not be able to know the flight ID of the first item, the flight ID of the second item, because this will change a lot, because it's dynamic. So, in this context, we are using this static one just to make it simpler, but keep in mind that this feature is super good
15:01
when this change. So, okay, the thing that we want to achieve here is to get dynamically these endpoints, to hit dynamically these endpoints, and not static. So, how can we do that? Okay, let's go again with the request. So, here is exactly the same stuff
15:22
that we have done before. We have tests here that, for this example, doesn't matter. But here is the different thing. We can store information inside the variables. So, here, let's grab the Pokemon name. So, response.json, then we get the results.
15:41
We have the results here, and then we grab the first element, this element here, and then we grab the name of it, Bulbasaur. So, now we are storing this information here using this variable. And if we want to use this in the next request, it will be available. So, now I can create a new request that is details, and then the method will be keeping being get.
16:03
But now I have the path that is Pokemon that uses this variable that was assigned before. So, concatenated it, we are going to hit the endpoint Pokemon slash Pokemon name, but with the variable value. So, let's see how it should look like.
16:21
So, now we are going to have the first one that is slash Pokemon, and then we are going to have the second one that is slash Bulbasaur. So, we could grab the information for a second request dynamically without putting like hard-coded Bulbasaur there. Okay, and how can I add SCAN API to my project? Here, I'm going to show you how we add the SCAN API
16:44
to our project as Load Smart, but since it's a CLI, so you can do the way you prefer. But this is the way that we introduced it in Load Smart, okay? But this is like for every step, you always need to do that.
17:01
So, you need to have a specification file and a config file. This is a config file just to set like some information, for example, the project name or to show in the report or just a configuration file. And here is the specification file. Okay, so in your project, you create a new folder that's called SCAN API, and then it will contain these two files.
17:23
And what do we do? We make this integrated with our continuous integration pipeline. So, every time that we have our API on master and deployed on stage, we hit the endpoints on staging using SCAN API.
17:42
Why we don't do that in the production? Because this is an integration test. So, it will interact directly with the database. We are going in fact to hit the endpoints. So, we cannot do that in production. So, the thing that we do, we do this after the deployment in staging and the report.
18:02
We start in the SQL CI artifacts so everyone can access it directly. So, here is how it looks like the configuration file for SQL CI. So, we use Docker image.
18:22
And then we have the steps to run the CLI and then we do the jobs. And we only do it after the push on stage. Okay, we have much more things like we have language independent. This is cool because it's like language independent.
18:41
So, it doesn't matter if your API is in Python and it's Ruby or it's implemented in Java. Since we are going to really hit the API, it doesn't matter how it's like cited. It also accepts JSON. We can use environment variable because for example, if you want to use a secret key,
19:01
you are not going to use to put it inside your GitHub project or your version codes. So, for that, we also have the, in the configuration file, you can set which information you want to hide in the report. To not show you, to show a secret key or something like this. We also have multiple file API specification
19:22
because this specification could be really big so you can split it. And yeah, we have custom templates. If you didn't like the template that we offer to you that is with this scan API logo and other stuff, you can implement one, showing the things that you prefer, just passing a Jinja template in the comment line.
19:43
Can I start using it for sure? We already have a stable version. We released it last month, so it's all good. We are already using it at Loadsmart. We have on our website that is still work in progress, but we already have a lot of information
20:01
that you can check there to use it. Next steps. We can implement more HTTP methods. We can improve our JSON visualization. We can have more docs, tutorials, improve our website, make the automation easier with GitHub Actions. But I think the best thing here
20:21
that we can think as next steps, like what if we could integrate the open API, like running an open API specification, and then it's generating and it's collecting off its scan API. And I don't know, what if we could integrate with DRF, Django REST Framework. So yeah, a lot of good steps.
20:40
Why to contribute? We have a lot of different stuff to contribute. So backend, frontend, automation design, we have a lot of different kinds of issue. It's per Python, so that's cool because we are not attached with any framework. It's nice to understand how a lib works, so how to deploy and pip, how to do, how to set up this lib.
21:04
We really care about quality of code, so we have coverage, good coverage. We try to label the issues with good first issue and all kind of these issues, labels. We have a lot of help from the component community. We try to create ADRs there to discuss
21:22
about design implementations. We try to make it super, trying to thinking together, not only one person. We are going to have a spring session. So on NeuroPython, so yes, if you want to join, please just join this channel on Discord
21:42
and have all the information. It will be during Saturday and Sunday. So please join us. And if you are here, you like the idea, but you're not going to use it and you don't want to contribute anyway, you can at least contribute into Givens starter. So it's github.scanapi.scanapi
22:02
and then you have the, we can, like we use this style to make it easier to share with information, like to get more visibility. We have organization here for scanapi with our apples. We have the Twitter, scanapi underscore. So follow us there. We are not so popular there yet.
22:21
26 followers, okay? And yeah, if you are interested about logistics, working for Loadsmart, here is a direct link or a care code. Join us, talk to me, come talk to me. And thank you very much. I want to thank you first of all for you that join and listen,
22:41
but for all the people from community that already help with scanapi, this is super, super cool project. I'm super happy with it. This is my Twitter, see my CD, my GitHub and here are the Discord channels to talk about it afterwards. So let's keep in touch. Let's keep in touch.
23:02
Thank you so much, Camilla. Thank you. Great talk. So let's see, we do have time for questions if anyone has any. Of course, the trick here is that my Zoom
23:21
is not updating one of the two screens to show things. All right, so right now we actually don't have any open questions, interestingly. So we'll hang out for a minute because if anyone has any questions, you know, put them into the Q&A here on Zoom, or if you need to, put them in the Microsoft track. But you know what, if you don't get any, Camilla, my professor in college used to say,
23:42
if you don't get any questions after a talk, it just means we're a really, really good presenter. I hope so. We do have one. So you don't have to go away empty handed. We also have a comment. So I'm going to do the comment first and then the questions. The comment, as Keith said, I actually have a work project where this would be super useful.
24:03
So. Awesome, I hope so. And then we have an anonymous question. How does ScanAPI handle authentication flow? Okay, so authentication flow you can use with environment variables. So for example, if you're using a JWT token, you can pass it as an environment variable
24:23
and then it's all good. So every authentication that depends on sending information via requests with secret keys will be handled. Excellent. And I don't see any others.
24:41
If anyone does want to chat. Oh, hey, as soon as I say that, somebody did post one. That's great. You also have to do is say there's nothing and then there's another one there. Anyway, the original asker says thank you. Someone else said this may be completely explained in the website. Haven't got a chance to check yet. Is there a timeline of things
25:00
you guys are looking at implementing? The open API integration sounds rad. So what was the question? Sorry, when we are intending to implement the open API integration? Michael asks, is there a timeline of things that you guys are looking at implementing?
25:20
Yes, I think first of all is the open API integration and yeah, first because we are already like for the load smart, it's already working fine. So I think we need to listen more from community what is missing and what is like
25:41
what people want more so to understand but the line of it, it's for sure like integrating with open API and probably something with the RF or something to make it easier to avoid people having to write the configuration by itself. All right.
26:03
And so the magic words again, I don't see anymore. Okay. It does look like that's it. So if anyone does want to chat more about this, there is a chat room specifically for this. So if you go to talk-scan-api over on Discord,
26:24
talk-scan-api and you can chat more with Camilla over there. So once again, thank you. Thank you very much for that. Thank you. Thank you very much.