Foreign API Simulation with Sinatra
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 82 | |
| Number of Parts | 89 | |
| Author | ||
| License | CC Attribution - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/31512 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
RailsConf 201682 / 89
1
2
3
5
7
8
11
12
19
22
28
29
30
31
33
45
47
50
52
53
58
62
67
71
74
76
77
78
79
80
81
84
85
86
88
89
00:00
Software developerComputer animation
00:35
Software testingSoftware developerTwitter
00:59
Core dumpCASE <Informatik>Software frameworkSpacetimeMobile appConnectivity (graph theory)Library (computing)TouchscreenSoftware testingSlide rulePoint (geometry)Spherical capProcess (computing)Connected spaceRuby on RailsComplex (psychology)Line (geometry)Type theoryIntercept theoremQuicksortEndliche ModelltheorieWeightSheaf (mathematics)Cartesian coordinate systemRemote procedure callCategory of beingGUI widgetSuite (music)Different (Kate Ryan album)Multiplication signGroup actionHookingAttribute grammar
03:39
Pattern languageCodeMobile appComputer virusCartesian coordinate systemDependent and independent variablesLambda calculusEqualiser (mathematics)Maxima and minimaStatement (computer science)QuicksortMessage passingEmailComputer animation
04:49
ImplementationDirection (geometry)Combinational logicQuicksortWeb 2.0Point (geometry)Mobile app
05:34
Axiom of choiceCartesian coordinate systemQuicksortBlock (periodic table)Mobile appEndliche ModelltheorieLibrary (computing)Social classParameter (computer programming)Replication (computing)Set (mathematics)CASE <Informatik>Computer animation
06:38
Combinational logicMobile appCartesian coordinate systemDatabase transactionResultantDomain nameEndliche ModelltheorieDependent and independent variablesLoginError messageRegular graphService (economics)Pattern languageLibrary (computing)QuicksortSoftware testingMessage passingComputer fileOrder (biology)Intercept theoremGoodness of fitInstance (computer science)Matching (graph theory)INTEGRALBlock (periodic table)System callWrapper (data mining)Computer programmingCASE <Informatik>Social classMappingMathematical analysisMereologyPoint (geometry)Right angleTheoryMeeting/InterviewComputer animation
11:56
CASE <Informatik>QuicksortComputer fileDependent and independent variablesSurfaceBitTwitterService (economics)Software testingPairwise comparisonInformationNatural numberMobile appDatabaseCartesian coordinate systemCodeSemiconductor memoryPower (physics)MereologySuite (music)Software developerInterior (topology)Sinc functionState of matter
14:42
Exception handlingSoftwareCartesian coordinate systemError messageLibrary (computing)Functional (mathematics)CASE <Informatik>Arrow of timeInstance (computer science)Computer animation
15:27
CodeStandard deviationDependent and independent variablesMiddlewareSuite (music)CASE <Informatik>Endliche ModelltheorieSoftware testingSystem callObject-oriented programmingHash functionPoint (geometry)Social classSet (mathematics)InjektivitätCodierung <Programmierung>Meeting/InterviewComputer animation
17:30
Power (physics)Exterior algebraQuicksortMereology
17:59
SoftwareState of matterConnected spaceOpen setService (economics)WeightVariable (mathematics)ImplementationDependent and independent variablesDatabase transactionMereologyInstance (computer science)Parameter (computer programming)System callLibrary (computing)Ocean currentMatching (graph theory)QuicksortPattern languageSoftware testingElectronic mailing listBookmark (World Wide Web)Suite (music)CodeArithmetic meanCore dumpCartesian coordinate system1 (number)Source codeCASE <Informatik>Slide ruleDefault (computer science)Block (periodic table)Server (computing)Keyboard shortcutRoundness (object)Point (geometry)Endliche ModelltheorieTelecommunicationWeb 2.0NumberInformationComputer animation
24:48
Different (Kate Ryan album)Error messageCycle (graph theory)Similarity (geometry)QuicksortCartesian coordinate systemArrow of timeCASE <Informatik>SoftwareWeb 2.0Computer simulationPropagation of uncertaintyMeeting/Interview
25:24
Product (business)Statement (computer science)Exception handlingCartesian coordinate systemQuicksortCodeError messageSoftwareNumberConfiguration spaceAsynchronous Transfer ModeArrow of timeComputer animation
26:04
PlanningAdaptive behaviorError messageDifferent (Kate Ryan album)WordMechanism designCASE <Informatik>QuicksortFactory (trading post)ImplementationSoftware testingMultiplication signFlow separationComputer virusComputer animation
27:10
Entire functionMechanism designSuite (music)Content (media)Service (economics)Right angleSlide ruleExterior algebraSoftware testingQuicksortDifferent (Kate Ryan album)Library (computing)Adaptive behaviorCodeComputer animation
28:43
Computer animation
Transcript: English(auto-generated)
00:11
My name is Konstantin Tenhart, and I have to warn you, this is not going to be a funny talk. I'm German. We don't do that.
00:24
So, but I actually don't live in Germany anymore. Last year I moved to Ottawa, Canada, and there's so many Ruby developers in Ottawa. I actually work for Shopify. And, in fact, we have so many Ruby developers that we just don't know where to put them,
00:41
and we sent them all down to Kansas to speak at RailsConf. And so Kat already gave a talk, and there's two of us speaking later in the afternoon about how we test and about sprockets. To continue with the shameless self-promotion, you can find me on Twitter and GitHub.
01:03
My handle is T60. And, in my spare time, I'm maintaining a couple of libraries that you might find interesting. One of them is ActionWidgets, which is a UI component micro-framework, I'd say, for Ruby on Rails. In fact, the slides you just see here on screen are powered by ActionWidgets.
01:24
I'm maintaining smart properties, which is supercharged Ruby attribute assessors, as well as a processing pipeline for Ruby on Rails to model complex business processes. And finally, the one I want to talk today about, RequestInterceptor, which is my most
01:43
recent one, and it allows you to simulate foreign APIs with Sinatra. So at its core, this talk is all about testing, and specifically one type of test, tests that involve HTTP connections.
02:01
So, most of you might know the libraries VCR and WebMark, which are usually used to stub out individual requests or, in case of VCR, replay requests that have previously been sent to a remote API.
02:21
I want to present a different approach today and talk about how we can use Sinatra to simulate a foreign API within our test suite. And that is essentially the core idea behind RequestInterceptor. So I guess I best show you how to use the library first, and then throughout the
02:44
talk we dive deeper and deeper into how it actually works internally, to the point where I'll show what kind of metaprogramming techniques I use to hook into the net HTTP library to make all that magic happen. So yeah, I just mentioned it.
03:02
RequestInterceptor does modify net HTTP, just like WebMark and VCR. There is no clean way to sort of interject yourself into what net HTTP does, so some trickery is required to make that work, but I get back to that later.
03:22
The idea is that you can use any rec-compatible app and use it as a HTTP request sent out by your application and reroutes it to your Rack app, which will then handle the request in line.
03:41
And in fact, all you need to know, essentially, is that RequestInterceptor implements a run method, which takes a Rack application as well as a hostname pattern. So the hostname pattern is important to know when RequestInterceptor sort of starts intercepting requests. It will actually look at the HTTP request and
04:03
only redirect the request to your own Rack app if it matches the hostname. Otherwise, the HTTP request will be made just as a regular remote request. And in the code example here, I define the probably most minimal Rack app you could potentially implement.
04:22
It's simply a lambda statement that returns an array with a status code, no headers, and a message, hello. And then I use RequestInterceptor to intercept all requests that go to anything that ends with, any host that ends with example.com. I do my HTTP request and
04:42
then assert on the equality of the response being hello. The problem with bare metal Rack apps is that they are very inconvenient. To sort of implement something more feature complete, you wouldn't necessarily want to go with Rack directly.
05:03
Instead, you wanna pick something that provides you with a little more convenience. And for me, this convenience is sort of given by Sinatra, which sort of combines simplicity as well as provides you with a lot of flexibility on how to simulate these API endpoints.
05:26
And for those of you who don't know Sinatra, it is a Ruby micro web framework, and it's based around a very simple idea. You have a Sinatra application that provides you with more or
05:42
less, well, the most important methods are get, post, put, and delete, which correspond to the HTTP methods. And they allow you to define request handlers in your Sinatra application. So they take a path as the first argument, and then a block, and the block defines how requests are being handled.
06:06
So a simple Sinatra app looks something like that. You don't even need to wrap it in a class or anything. It provides you with some magic to make this work, and you require the library. You define that your application is handling anything that comes into
06:23
slash hello, and in this case, it returns hello Sinatra. So given this conciseness and this simplicity, Sinatra was an excellent choice to sort of model APIs. And therefore makes a great combination with RequestInterceptor.
06:46
In fact, I went further because of this great combination. It's the combination I would suggest for you to use instead of using RequestInterceptor.run. With just any Rack app, I would recommend using Sinatra.
07:03
And RequestInterceptor gives you a define method, which allows you to define a new Sinatra application with some extra goodness. So the RequestInterceptor allows you to define the host name pattern.
07:22
Again, just as we've seen before, where we submit the host name pattern and the application to the run method, we now define it right on the application. And then we just define it as a regular Sinatra app with all of our endpoints that we need.
07:41
And the result of this define call is a class again, which is a Sinatra application with the added benefits. And one of those benefits is that this application provides you with an intercept method. And the intercept method is just a convenient wrapper for you around run.
08:00
So instead of having to pass in everywhere where you want to use an interceptor, remember which host name you wanna match and which application to pass in, you can just call intercept on your interceptor, provide it with a block, and then again, fire off an HTTP request. And assert that the correct message is returned.
08:25
And then more importantly, in order to test this, you probably want to know how many requests you made, which requests you actually made, and what the request and response data was. And to make this possible, the intercept method returns a transaction lock.
08:45
So it's simply an array of RequestInterceptor transactions. And these transactions are simply structs, which give you access to the request and response that was made within the block.
09:02
And these are instances of Rackmock request and Rackmock response, just as other libraries usually use for testing Rack applications. I essentially use these to carry all the data for further inspection. And then the example down below shows you how you can, for instance,
09:22
assert on the path of the first transaction log entry. And in this case, I'm just asserting that my program called the path hello of example.com.
09:40
You can also nest them in case you communicate with multiple APIs. And at Shopify, I was on the team that implemented Uber integration. We did that as a separate app. So for us, we also treated Shopify as an API,
10:02
just as you would if you develop an app for Shopify. And then we treated Uber as our other service. So our application was actually had to communicate with both of these services. And it's often necessary that you know exactly which requests were sent where.
10:22
And that is why request interceptors do support nesting. So both of these interceptors write a separate transactional log. And yes, of course, the innermost interceptor takes precedence. So you can actually have two interceptors responding to the same domain or
10:44
to the same host, in which case the innermost would win and intercept the request. Another important feature is that you can customize an interceptor for an individual test because the idea is that you generally outline your service
11:03
that you are modeling in a single file. And then customize it to certain behavior that fits sort of the needs of your test. Let's say you wanna model an error response for one particular endpoint.
11:22
You would take your interceptor, call the .customize method on it, and then override the previously defined endpoint. And Sinatra is smart enough that if you redefine an endpoint, the new endpoint will take precedence over the old one.
11:42
And in this case, we are just switching the hello endpoint to send another message. Previously, it was hi rezconf, and now it's bonjour rezconf. So now that you have a basic understanding on how they work,
12:01
I wanna talk a little about the advantages in comparison to VCR and webmuc that I think exist when using request interceptors. For me, one of the biggest advantages is that the code isn't cluttered throughout your test suite. Instead, what we do is we have one file that defines a particular service,
12:25
in our case, Uber or Shopify, that implements all the endpoints we are usually communicating with. And then we customize this interceptor to specific needs in our test suite. But if you sort of wanna see in one go what your app is actually
12:43
communicating with, you would just open the file and look at the interceptor definition. Another advantage for me is that interceptors provide greater power and flexibility because we're talking about a Sinatra application.
13:03
You can literally go as far as you want with that. You could have theoretically an inner memory database that sort of keeps state if you wanna simulate entire workflows or you can keep it super simple and return static responses from your endpoints.
13:22
So it's really up to you. Then, of course, since it's essentially just one file, you can also go further and package it into a Ruby gem. Let's say you build a service that other developers use and you have a public API and now you want to make it easier for
13:43
people to sort of integrate your service. You could provide them with a predefined interceptor they can use in their test suite. So they don't even think about hitting your API with requests from their test suite.
14:01
And then finally, and that is personally for me super important, is that the code is just very readable, which is in the nature of the Sinatra application. And I personally think it's more readable than having these VapMark stubs sort of scattered around your test suite. Instead, you have this one single application that defines how
14:23
your interceptor works. And then there's more. There's features that I am not sure if you could simulate them with VapMark or VCR. And so I wanna talk a little bit about more advanced concepts on how to use these interceptors.
14:43
A big one for me is simulating network requests. Recast interceptors are set up in a way that they propagate errors or exceptions that are being raised in one of the endpoints. So I specifically disabled Sinatra's functionality to handle exceptions.
15:04
And propagate them through the entire stack, which allows, for instance, to simulate that a host is unreachable, simply by raising the appropriate exception. Which makes it very easy to test your application or the library you're building, whether it's robust enough to handle these
15:23
error cases. And then of course, Sinatra gives you a lot of tools that you can leverage to make interceptor definition even easier and make the code more readable. And one of the most important things is probably that being a standard Ruby class,
15:44
you can just define private helper methods that you can use throughout your interceptor and throughout the customizations you use in your test suite. In fact, you can just apply standard object-oriented design principles to and
16:01
all that Ruby gives you to sort of make your interceptors as readable and as easy to use as possible. Then there is the possibility of using Sinatra's before and after callbacks that run before or after a particular endpoint is hit.
16:23
And you could, for instance, utilize an after callback to automatically encode data into JSON, let's say you're modeling a JSON API. It's tedious if in any endpoint you always have to remember that you, as a last step, have to call to JSON on whatever you're sending over the wire.
16:42
So just define it once in a block, and in this case, I look at the response. And if it's an array or hash, I encode it into JSON. And then of course, you have the ability to use Rack Middleware.
17:00
And in this case, we modeled both Shopify and Uber interceptors as JSON APIs. And so we always wanted to decode the incoming JSON, so we can easily work with that in our interceptors. And Sinatra provides you with a method called use that allows you to
17:21
inject Rack Middleware that runs before your actual endpoint is hit. Now that you have sort of an understanding on how you use interceptors and why they might provide a nice alternative to VCR or WebMOC,
17:42
I actually want to dive deeper into some of the internals. Because I just think it's interesting to see some of the powerful features Ruby provides, and just as a sort of learning exercise. So in the beginning of the talk, I showed you that a request interceptor.run
18:05
is sort of the core of the whole idea. And in fact, this is the concrete method implementation as it exists in the library, and there's essentially six steps. And I will go over all of these six steps to sort of showcase how you can
18:24
mess with an existing Ruby library that doesn't provide you with the ability to sort of do this in a clean way. So the first step is, because you can reuse an interceptor, is to clear the transaction lock. That's very easy, I just clear the array that
18:42
keeps all the transaction lock entries from the previous run. And then I cache the original net HTTP methods because we have to make sure that once the block finished its execution, we restore net HTTP to its default behavior. And then I override the net HTTP methods with a custom implementation,
19:04
just as WebMark does as well. And then I execute my test, and now my test will essentially use these overridden net HTTP methods. And then finally, I collect my transactions and then eventually restore net HTTP to its former glory.
19:24
And the last part happens in an ensure part. So it's always guaranteed to run, and so that it doesn't happen that your test suite actually gets into a state where
19:41
net HTTP is not in its original state. So as I said, it's easy to clear the transaction lock, so I just wanna skip that and talk about caching the original methods. There's three methods you need to override if you wanna
20:02
do something like incepting HTTP request. There's start, finish, and request. Start and finish sort of take care of opening the TCP connection, and then request performs the actual heavy lifting. And the way caching works in request interceptor,
20:21
you have now a concrete request interceptor instance at your hand that is currently handling your test case, and I just assign these methods to instance variables. And what instance method gives me is an unbound method. So I essentially save the original method implementation and
20:42
just put them for now in an instance variable. And then I replace these three methods with my own implementation. Start and finish are pretty boring. I just make sure that net HTTP thinks it has an open TCP connection. It is communicating with, but in fact, I don't need one because of how
21:04
the redirect to the Synapra application is working. And I'll show that in a second. And then I define a new request method, which is a little more interesting. The interceptor instance itself that is currently handling your test case
21:20
has a request method of its own. And all I really do is I take the data that would usually go to net HTTP request and redirect it to my interceptor. And then I also pass in the interceptor itself. I won't show the code for request interceptor request,
21:45
because it's a little more complex, but I at least want to explain what is going on. And you can always take a look at the source code if you're interested. So the first thing I do is I try to find an appropriate interceptor, meaning I look at the HTTP request and
22:03
then look at the host name of this request. And now go through my list of host name patterns and store the applications and see if one matches. If I find one, I now build a mock request. And mock request, the initializer of mock request,
22:20
takes a rack application as its first argument. Once I have that mock request initialized, I can call the methods get, post, put, delete on them to simulate an actual HTTP transaction. And once that happened, I get back a mock response,
22:41
which I now have to transform into a net HTTP response to make net HTTP believe that it actually just talked to remote service. And then I log the transaction, meaning now I'm taking the mock request and the mock response and just writing them in my transaction log so
23:00
they can be further analyzed in a test suite. The interesting thing is what happens if no interceptor actually matches your host name, because I wanted to implement it in an unobtrusive way. I didn't want it to block just any HTTP communication, especially to be still compatible with MacMock and VCR.
23:24
So what happens is my current net HTTP instance, which is now in this weird state that it talks to this natural application, has to be restored to actually be able and perform network requests.
23:43
And the way I do this is shown on the next slide. But once I restored it, I essentially perform the request as if there would never have been any interceptors in the way. And method restoring works by utilizing Ruby's defined method.
24:04
We actually can not just take a block, but it can also take an unbound method. So the ones we previously stored in instance variables, we can now rebind to net HTTP.
24:20
And we can even rebind them to concrete instances of net HTTP. And it is sort of happening when the request interceptor doesn't find the matching application. It rebinds the original methods to the concrete net HTTP request that's currently going on, and then just calls request again and
24:40
performs the request as if nothing ever happened. So that was essentially the internals of how the request cycle works in request interceptor. And if you compare that to WebMark, there's certainly similarities
25:03
with the difference that you define a stop within your test. And in this case, I redirect to the Sinatra application. I previously mentioned that there's error propagation, that you can utilize to sort of simulate network errors.
25:22
And I just wanted to quickly show how this works. It is very simple because Sinatra supports it by just using particular configuration statements. So all you need to do to sort of have a Sinatra application actually raise an exception and not handle it, and
25:41
have the calling code take care of that exception. Is you disable the show exceptions, and you enable raise errors. And by that, you sort of switch Sinatra into an aggressive mode, which does not make sense if Sinatra runs as your production application. But it makes a lot of sense to sort of simulate these network request errors.
26:07
Well, I do have further plans for request interceptor. So one thing I wanna implement is the support of traits. Sort of similarly named like the factory girl mechanism,
26:23
where you can define what your factory is building, and then give it a certain trait of how it is actually building. And I want that for interceptors as well, because I was running into the issue that I was simulating the same end point several times.
26:40
And what I did so far was just having a lot of these customized request interceptors. But what I actually want is just in a particular test case, I wanna have a name where I can refer to an end point definition and say I want my interceptor to run with a faulty implementation of my hello.
27:01
And the faulty implementation could either be raising a 500 or raising a network error. And I wanna support different adapters. So I don't wanna just stop at net HTTP. The next thing I wanna implement would be Faraday, because Faraday would give me exposure to several other libraries.
27:22
Because I don't really wanna mess around with each of these libraries individual. That is sort of the two goals I have in mind right now, to bring this library forward. And that basically brings me to the end of my talk. And I just wanna quickly summarize what I've been talking about.
27:45
So request interceptors sort of provide a third alternative to VCR and WebMock. The thing I like most of them is that I have a concise service definition in one place instead of scattering this definition across the entire test suite.
28:02
And they provide me with an easy mechanism to customize them if there is the requirement in a certain test. And then finally, Sinatra provides me with a lot of simplicity and flexibility, which ultimately leads to very readable code, which is just something I greatly enjoy.
28:23
If you're interested to take a look at the slides again, because I know it was a lot of content I was going over. They are available online. Thanks a lot for your attention.