Building mobile APIs with services at Yelp
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 12 | |
| Number of Parts | 173 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/20208 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Place | Bilbao, Euskadi, Spain |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
EuroPython 201512 / 173
5
6
7
9
21
27
30
32
36
37
41
43
44
45
47
51
52
54
55
58
63
66
67
68
69
72
74
75
77
79
82
89
92
93
96
97
98
99
101
104
108
111
112
119
121
122
123
131
134
137
138
139
150
160
165
167
173
00:00
Executive information systemConnected spaceMobile WebWebsiteMetropolitan area networkWeb pageHaar measureLocal ringConnected spaceSlide ruleQuicksortFigurate numberSource codeDigital photographyMeasurementMobile WebWeb pageMobile appXMLUMLLecture/ConferenceProgram flowchart
01:27
Front and back endsSoftware developerSineCodeRepository (publishing)World Wide Web ConsortiumMobile WebCodeSoftware developerMobile appMedical imagingRepository (publishing)Cartesian coordinate systemCode refactoring1 (number)Front and back endsStatement (computer science)AreaMobile WebAbstractionSystem callTemplate (C++)Point (geometry)SequelJSONXMLUML
03:19
Metropolitan area networkCodeXMLUMLComputer animationLecture/Conference
04:47
CodeRepository (publishing)World Wide Web ConsortiumFront and back endsMobile WebPhysical systemLevel (video gaming)MathematicsProduct (business)Process (computing)Physical systemCore dumpSuite (music)Level (video gaming)Branch (computer science)Formal verificationSoftware testingCodeDigital rights managementOpen sourceRevision controlData centerBitMedical imagingAbstractionMultiplication signMassScaling (geometry)Flow separationBound stateXML
08:27
Port scannerMagneto-optical driveSoftware developerWeb serviceIndependence (probability theory)CodeMetropolitan area networkInterface (computing)Graph (mathematics)NP-hardConsistencySoftware testingControl flowMathematicsIntegrated development environmentMathematical singularitySystem callCodeSingle-precision floating-point formatWeb serviceMathematicsProduct (business)Level (video gaming)Line (geometry)Software testingInsertion lossMereologyDependent and independent variablesSubsetFlow separationNumberMedical imagingControl flowMultiplication signDomain nameInterface (computing)Electronic mailing listGraph (mathematics)Confidence intervalSystem callResultantSoftware developerData miningChaos (cosmogony)Closed setFunctional (mathematics)Set (mathematics)Unit testingConsistencyProcess (computing)Connectivity (graph theory)SoftwareService-oriented architectureWave packetIntegrated development environmentExecution unitMobile appDatabase transactionUniversal product codeLecture/ConferenceJSONUML
13:43
Maxima and minimaMetropolitan area networkGamma functionWeb serviceWeb serviceFile formatConfiguration spaceGame controllerRule of inferenceControl flowSet (mathematics)Link (knot theory)Arithmetic meanData storage deviceRevision controlDebuggerBitSynchronizationSoftware testingCodeRight angleSubsetCommunications protocolFlow separationDifferent (Kate Ryan album)JSON
15:04
Web serviceSystem callWeb serviceData modelHypermediaRandomizationParameter (computer programming)Block (periodic table)Endliche ModelltheorieGame controllerSpacetimeDependent and independent variablesSoftware frameworkXML
15:51
Metropolitan area networkDensity of statesDependent and independent variablesWeb serviceData typeClient (computing)TypprüfungComputer multitaskingoutputType theoryOpen setParameter (computer programming)BitWeb serviceDependent and independent variablesSet (mathematics)SoftwareLibrary (computing)Mobile appSystem callField (computer science)Power (physics)Functional (mathematics)Data modelData structureClient (computing)Android (robot)CodeValidity (statistics)Revision controlSinc functionNumberSummierbarkeitPoint (geometry)IRIS-TOpen sourceConstraint (mathematics)Computer animationJSONUML
17:50
Web serviceMedianAreaSineProxy serverExecutive information systemSystem callMathematical singularityFreewareData modelDifferent (Kate Ryan album)Maxima and minimaWeb pageClient (computing)Interface (computing)Level (video gaming)Front and back endsData acquisitionMobile WebSoftware testingImplementationMetropolitan area networkWorld Wide Web ConsortiumServer (computing)AreaClient (computing)Mobile appData storage deviceWeb serviceFile formatProxy serverWeb pageSystem calloutputMultiplication signMathematicsAndroid (robot)Set (mathematics)Level (video gaming)Mobile WebSoftwareWeb 2.0Theory of relativityRevision controlServer (computing)Arithmetic meanImplementationWeb-DesignerCASE <Informatik>Software developerTerm (mathematics)Point (geometry)Integrated development environmentBitLecture/ConferenceComputer animation
21:31
World Wide Web ConsortiumRevision controlPort scannerMultiplicationRevision controlDependent and independent variablesClient (computing)Multiplication signComputer animation
22:02
Revision controlPort scannerMultiplicationDependent and independent variablesServer (computing)MereologyField (computer science)Validity (statistics)Computer animation
22:35
Revision controlPort scannerMultiplicationOpen setTask (computing)Server (computing)Queue (abstract data type)Web serviceLogical constantOnline chatPosition operatorNewton's law of universal gravitationBlogGamma functionMetropolitan area networkType theoryMathematicsMultiplication signDescriptive statisticsString (computer science)Point (geometry)Computer fileRevision controlField (computer science)Task (computing)Web pageMetric systemSoftware developeroutputError messageVideo gameComputer hardwareMobile appAndroid (robot)Software testingClient (computing)Open sourceSound effectIntelligent NetworkCartesian coordinate systemWebsiteNetwork topologySource codeBit rateData analysisCrash (computing)Coma BerenicesBlogWordOffice suiteWeb serviceNumberRotationProduct (business)Process (computing)TwitterIdeal (ethics)Server (computing)JSONXMLUML
26:06
Executive information systemReal numberWeb serviceSubsetSoftware developerMobile appType theoryMathematicsSuite (music)DivisorDifferent (Kate Ryan album)Android (robot)Data structurePhysical lawSoftware testingCartesian coordinate systemVolume (thermodynamics)CodePoint (geometry)outputBranch (computer science)Revision controlProcess (computing)InformationServer (computing)Dependent and independent variablesLatent heatProduct (business)Overhead (computing)Control flowMultiplication signLengthCoefficient of determinationGoodness of fitModule (mathematics)TunisData exchangeEndliche ModelltheorieSystem callUnit testingSource codeModal logicPlanningConnected spaceWindows RegistryQuicksortMusical ensembleBitInterface (computing)Client (computing)Local ringState of matterOpen setVirtual machineGraph coloringOrder (biology)Similarity (geometry)Variable (mathematics)Web 2.0MultiplicationPresentation of a groupWeb pageLecture/Conference
Transcript: English(auto-generated)
00:05
Thank you and sorry for the flicker, I hope you can manage to see the slides anyway. Who here knows what Yelp is? Oh, that's actually, sorry, actually quite a few people so I'll be quick.
00:22
Well, Yelp's mission is to connect people with great local businesses. We have a website, an app and a mobile website. We have 142 million unique active visitors monthly. We have 77 million reviews by those users and we are available in over 30 countries.
00:45
We most recently launched in the Philippines and we're not only for finding like great restaurants and bars but also like great doctors, great shops and any other kind of local business. We also have, this is probably like less known, we have Yelp for business owners.
01:03
So if you're a business owner you can come to Yelp, you can claim your business, you can market as your own and you can then measure visitor activity on your Yelp page, you can interact with customers. So if customers leave a review for your business you can reply to that in public or in private
01:21
and you can upload photos for your business and do a bunch of other things. So who am I? I'm a back-end developer for the BizOwner app. I worked on the main Yelp app back-end before that. I'm a Python user since 2008. I started doing a lot of Django work back then
01:42
before switching to application and now mobile development. So let's take a look at why we're going to, okay, that's already not good. There's supposed to be an image there about Yelp. Let's take a look.
02:02
We were founded in 2004 and actually like one of the co-founders, Jeremy Stoppelman, is still running the company. He's our CEO. And all of our code was in a central repository we called Yelp Main, which means the code for the website including templates, the mobile web, the mobile back-end
02:23
and the business owner side, all in one repository, which means we had a lot of homegrown code. And as people worked on it and they introduced new abstractions, they didn't remove the old ones and it was hard to reason about the code,
02:40
do these big refactoring. So as Yelp grew and we are still growing, we're still hiring people, this started to become a bottleneck. We actually, like at one point, we had three different ways to do SQL statements, or execute SQL statements in Yelp Main. That was not nice. So yeah, we cannot really refactor all the code and I want to dig deeper into another area
03:06
that really highlights our bottleneck. I'm really sorry, like, I think I need to put this, really sorry about this.
03:51
Okay, yeah, sorry about all of this.
04:47
Yeah, so we had a lot of homegrown code. Finally we see the images, this is what Yelp looked like back then. We have a lot of abstractions, I talked about that. So let's talk about the push process, which is what we call when we deploy Yelp code.
05:06
We do deploy Yelp code several times a day. This is done by a Pushmaster, which is an engineer that has production system access. People take their code changes, their code review changes,
05:20
they want to push to production and they join a push, and then the Pushmaster runs this push. We have several tools to assist us in doing this. You can see a screenshot of Push Manager, which is actually open source, where we manage the pushes and people can say, hey, I want to join this push and I want to push my changes to production.
05:43
And as you can hopefully see, it's not that clear, there's like a small red bar next to my push. I ran and it's red, which means this push didn't make it to production. We had to abandon it and I'm going to talk a bit about why this might happen.
06:03
So when we run a push, at first there are some automatic checks that take all changes, build a deployment branch where they merge all the changes in. That deployment branch is then deployed to a stage system. And then after manual verification, so all people that join this push,
06:22
they need to be present, they need to verify that their changes work on the stage system. And if everything is okay, we are happy with our test suite runs, then we send this branch to production. We do the same thing, we watch production for a certain amount of time. And if we are happy, the push gets certified, the changes get merged into master
06:44
and starting from that, the changes are live and we're done. People, when they branch off and they start to work on a new change, they will branch off of these changes. This is a two-hour process with really no upper bound.
07:01
Why no upper bound? Because like if we find problems, let's say on the stage system, we need to take out the problematic change, rebuild the deployment branch, put it again on stage system, run our test suites, go again to production and so on and so on until we have a new code version that is good and that we can leave on production.
07:22
You see here a screenshot, this is actually like another tool that helps us during the push where you can see which hosts in our data centers are already running the new version of the code. This is the green bars. Which hosts in the data centers are running the old version of the code, which is the red bars and the yellow bars are the hosts that are in the process of bouncing,
07:45
so switching to the new code version. I have to say our release engineering team, they are hard at work in optimizing this process. They're really making it better and better and more automatic, but still it was obvious that this doesn't scale.
08:03
You can run only so many pushes a day and the more people join Yelp and we are still growing, the harder it becomes to have a push without issues. Some intelligent minds sat together at Yelp and they thought about a solution and they found one.
08:23
I don't think we are the first company that came up with this solution, but it's kind of obvious we need to modularize. We are at a certain size where you cannot work with a single code base. You can only run so many pushes a day, as I just said, and even if you increase the number of pushes,
08:43
the number of people that develop at Yelp also increases, so you will run into problems eventually. Let's build services. How do services solve this problem? Each service is developed and deployed independently, so you don't actually need to know about this huge code base.
09:04
You just need to know about your service. Service pushes are very easy and very quick. People can do it themselves after short training. Services usually only cover one aspect or one set of features,
09:22
which also makes it very easy to introduce new technologies, to refactor code, reduce technical debt, all that kind of thing. And actually it might even bring some performance benefits, since when you have this big monolithic code base in Python,
09:41
it's not that easy to parallelize things, right? So when you switch to a service-oriented architecture, even though at first you might think, hey, I'm doing network requests instead of function calls, this should be slower, it might be actually faster than before if you do those requests asynchronously at the same time
10:03
and just wait for the longest result. We actually also wrote like Yelp service principles. It's like a list of do and don'ts for services or reasoning about services or thoughts. Go check it out. It's on GitHub.
10:23
Yeah, so why might we not want to do services? Because, well, chaos might ensue. I actually stole this from Fred Hatful, a colleague of mine. He also gave a talk about services. You can find it online. It's really good. Well, why not services?
10:43
Consistency is really hard. It's actually non-existent. There is no such thing as a transaction over several service calls. You don't have clear dependency or usage graph, which means you need to maintain your interface with your API forever since you don't know who is going to use it and for how long.
11:04
It also means that testing one huge self-contained code base, it's easy. It might not be simple, but it's easy. But how do you test your loosely coupled services which are out there? So this is the chaos I was talking about. How do you make sure stuff doesn't break?
11:23
Unit tests, everybody loves unit tests, or at least many people do. But, in my opinion, they are great, but they are not enough since in a world of loosely coupled services, breakage many times occurs at the interface level. So like some service you call, it subtly changes its interface, its API.
11:46
The developers maybe didn't even intend to, and yes, your call breaks. The answer is not that what you expected it to be, and this is a huge problem. What's our solution to that? Our solution is acceptance tests.
12:02
So instead of mocking, like with unit tests at the function level or further out, we don't do any mocking. We run all the code from the request as it comes in, through all services we might call, back the whole workflow, and then we test the response we get and make sure it's what we want it to be.
12:23
It's as close to production as possible without setting up your own dedicated stage environment. And it's what we do at Yelp. So we put all the components, the services, anything else you need.
12:40
In Docker images, we spin up those Docker containers, we use production code for these containers, and we use Docker Compose, it was previously called Fig, to manage these infrastructures. It's a bit heavy weight, so it takes a long time to run. Not the test itself, that's actually fairly quick,
13:01
but spinning up all those Docker containers, setting everything up, that takes quite some time. And it actually grows with the number of services you have, obviously. So you call more services, so your acceptance testing setup has to grow accordingly. So it's a bit heavy weight, but we're really happy with the results,
13:25
since it gives you a certain amount of confidence in your changes, because you can say, yeah, this is going to work in production. So, just an example of what this might look like. This is part of the acceptance testing setup for the BizOwner app backend,
13:44
so we have some configs, we have the main BizApp definition, where you can see under links, this is all the dependencies we have, all the direct dependencies. Those are several different services within Yelp. Internal API is actually the service front end of Yelp main,
14:03
so that's our legacy code base. And you can see on the right, that one itself has a bunch of dependencies, so that's how your acceptance testing setup grows. Yeah, but as I said, it can be a bit cumbersome also, like setting up test data, because some services have their own data store,
14:23
so when you create your test features, you need to make sure all services are in sync, have the same data, so your tests work. But overall, we're pretty happy with it. So now that we know why we do services and how we make sure they don't break randomly,
14:40
what's our service stack? We originally started with Tornado, but that did not work out quite as well as we hoped. So our current stack is Pyramid, just the latest version of Pyramid, with uWhiskey and SQLAlchemy, and it works out quite well. We use HTTP obviously as transport protocol, JSON for the data format,
15:05
and one very important block of our service stack is Swagger, which is an API framework, so with Swagger you specify your API, you write actually JSON to specify your API, and there's a bunch of tools included.
15:21
One of them is Swagger UI, which helps you visualize the API you just defined. This is what it looks like for a random service at Yelp. I think this is business media. So you see your methods get and post, you see the endpoint, you see the request parameters your service expects,
15:42
or that endpoint expects the data model you will get as a response, so you can browse and find all the endpoints you might need for your work. Swagger does more. It also does request and response validation optionally, but I would encourage you to activate that,
16:03
since it makes sure your request parameters are there and in the specified type, as specified in the spec, your response actually fits what we saw here in the data model. It does data structure and basic type checking on individual field level,
16:23
and it works dynamically by reading a service's spec. So there's a library called Breveto, it was called Swagger Pi, it's open source, it's on GitHub, on our GitHub account, and it dynamically reads the spec and generates your stub,
16:41
so you can do function calls or method calls in Python, which actually do the HTTP requests, and we used to do that with client libraries, which was quite painful, so say you wanted to develop a new endpoint for your service, you would do that, then you would check out the client library,
17:01
you would generate the stub code for that new endpoint, you would commit that after it went through code review, you would bump the client library version number, and only then, when people upgrade it to the new client library, they could use your new endpoint. All of this, Swagger Pi or Breveto, takes care of it for us,
17:21
and it makes it really nice to work with it. So let's talk a bit about a specific service, the BizApp service, which is the service that powers our BizOwner app clients, Android and iOS clients. It's a bit of a special snowflake since it's one of the very few services at Yelp you can reach from the outside,
17:42
usually you can only reach them from the internal network. It's also, unlike other services, it's not constrained to one set of features or one area, it contains the whole API for our app clients, and it has no local data store.
18:01
So actually many services have their own data store, we don't, so oftentimes we are just a proxy, we are calling other services, we are calling Yelp main, we are aggregating data and returning, formatting it, enriching it, and returning it to our clients. So how does our mobile API look like?
18:21
Well, it's a REST API, right, one resource per endpoint, do multiple calls to fetch related resources, blah blah blah, you probably already know all of this, and this is how we develop services at Yelp, but not how you do a mobile API. Because you are over a cellular network,
18:41
you want to be as efficient as possible, and you want to do as few calls as possible. So what we do is, we have one endpoint per client app page. So for every page your app displays, we just want it to do, if possible, just one network request and send it all the data it needs.
19:02
For post endpoints, whenever you want to save something from the client, we not only acknowledge that the write happened successfully, but we also send the client back all data it might need for follow-up pages to display that. This is quite different from our lower level service APIs
19:21
that really are more RESTy. So you can say that we aggregate, we do like many service calls for, typically many service calls for one client request we get, and we just aggregate and send data back and act as a proxy.
19:41
So what does it mean to develop a mobile app backend? I come from web development, I imagine some of you do as well. Turns out mobile apps have releases. In our case, for the BizOwner app, they are synchronized, so we release Android and iOS at the same time
20:01
with the same set of features. And iOS apps need to be reviewed, as you might have heard, and this actually takes quite some time. I remember it used to be like five days. Nowadays, I think our longest review time was 11 days. I think it's back to about nine days. That's actually like quite a long time,
20:22
and you probably want to test the whole thing before you release it, and in this case, releasing means submitting it to review to Apple. So our API needs to be done sooner than the client implementation, and which means it needs to be done way sooner than when the app is released or when you can download it on your phone.
20:44
How else is it different than web development? You cannot upgrade the client whenever you upgrade the server. In fact, some clients never upgrade. Like, we still have a tiny portion of users on the 1.0 release
21:01
for the BizOwner app, which we released late last year. So unless you want to drop support for those users, you need to support your API forever, which means you cannot do backwards incompatible changes. How do we deal with that?
21:21
We do a multi-version API. So we have the same endpoint with a different version. In this case, we append the version at the end, and we do maintain and test all versions. to make sure they still work. This is obviously, it costs something, it costs effort.
21:42
So maintaining multiple versions, we don't want to do that needlessly. So what are the ways we can think of to make sure we don't have to do multiple versions all the time? It turns out, if you just add data to the response, that's backwards compatible. Our clients, they will just ignore it, the legacy clients, the old clients.
22:02
And our response validation, what Swagger does for you, it's also smart enough to just ignore additional data. Just make sure that the data, as it's defined in the spec, is there. And once we develop on the server and we add that new field, we also add it to the spec.
22:21
So actually, our response will be okay, it will validate, right? So this is what it looks like. This is an example out of a JSON spec for an API. And you see the green part, we just added one field, time zone, type string with a description. And we could do that without having to do any other change to that file.
22:44
So obviously, like we didn't introduce a new version endpoint, it would just work. So how do we make sure it actually does work in production? Well, we do some monitoring. We monitor the number of requests, the server errors,
23:02
task queues, send push notifications. Here are some examples. This is a tool, like an older tool, you can almost not see it, where we look at the types of errors that happened at the rate of errors. We now have a bunch of nice Kibana 4 dashboards where you can actually do almost anything you want.
23:22
We send a bunch of our metrics to signal effects so you can build nice dashboards. You can visualize them, analyze them. And we use Elast Alert, which is a really nice open source tool. We open sourced, I think, last year. You should really check it out. It's very easy to set up your own alerts
23:41
so you know whenever something is wrong. For app crashes, so whenever our client apps crash, we use Crashlytics, both on Android and iOS. And as soon as you reach a certain size, you probably need an on-call rotation, so you need to wake up people whenever things break.
24:02
We use PagerDuty for that. We have integrated Elast Alert into PagerDuty, so for severe errors, we get paged. We have integrated Crashlytics, so if our app crashes spike, we get paged. Yeah, that's basically already about it.
24:22
I want to just mention another talk. If you are interested in services, Scott Trillia is going to give another talk about services arrested development, surviving the awkward adolescence of a microservices-based application. That was hard.
24:41
It's Friday, 11 a.m. in the Python Anywhere room. Go check it out. It's really a great talk. Also, some other shameless plugs. We are hiring, so if that sounded interesting to you, check out yelp.com slash careers, and even if you don't find your ideal job opening,
25:01
contact me or contact us on our booth. We will figure something out. We are always looking for talented people. We actually have offices in Hamburg, Germany. This is where I work. Also in London and obviously in San Francisco. Yeah, we also have an engineering hub
25:20
where we aggregate our blog posts. We have our open source efforts documented there and more. We are on Twitter. And last but not least, this is a fun one. I urge you to check it out, the Yelp Dataset Challenge. If you ever wanted to do some data analysis but you didn't have data or you didn't know what to do, go look that up.
25:41
The last one just ended, but the new one will probably start before the end of summer. Go check the website. It will be announced there. And the deadline will be sometime by the end of the year. It's a lot of fun and you might actually even win some money. So, that's all. Thank you very much.
26:02
Sorry for the technical difficulties. If you have any questions, just ask.
26:21
Can you talk a bit more about your aggregator? Is it something open source? Do you plan to open source it? Or is it a process on its own? Or is it a web server module? How does it work? Aggregating what? For your APIs. You say you have an aggregator for making only one request
26:45
and one request per page and not ten requests. That's basically what our BizApp service does. So, when a client, an app makes a request, an HTTPS request to our service,
27:02
it hits our servers, it hits our service and our service does everything it needs to satisfy that request. So, it will do multiple service calls, aggregate the data, collect it from different services, from Yelp main, put it together, fetch related data, everything the client needs,
27:21
and then send that back to the client in JSON over HTTPS. So, actually like our service, what we develop is the aggregator and does all of this. And sometimes when we want to aggregate the data, there is, like I mentioned, this internal API interface into Yelp main.
27:41
Sometimes there is no interface for the data we need. So, actually we will also be developing that interface and then use it to fetch and aggregate the data. I have actually two questions. The first one is what was the problem with Tornado?
28:04
Just curious. Yeah, I was fearing that issue might come up. I was actually not at Yelp when this was tried out. You might try to ask Scott about it. I don't know if he knows. You can also come to our booth. There are other people we can ask.
28:21
Honestly, I cannot tell you. I just know that it didn't work out well and now we are really happy with Pyramid. Okay. And the second question is how do you handle logging in the Docker containers? Yeah, that is an issue. We do logging. It's done inside the Docker containers. We expose the logging folders as volumes.
28:44
So, you can actually start another Docker container and mount those volumes from the individual Docker containers and look at the logs if necessary. Hi, thanks for the talk.
29:02
Can you comment more on your development process? Because I can imagine that testing is a huge overhead. So, do you have a separate team that makes testing or does every developer is able to set up the whole infrastructure with Dockers?
29:22
How does it work? Thanks. Yeah, that's a great question. Yes, every developer is supposed to not only write the code but also write the test. So, typically like the development processes, you create a branch from within Git, right? You do your development. Once you're ready, you post your changes for code review.
29:44
Other developers review the code and hopefully if they pay attention and you did a change without adding tests for that change, they will say, hey, you should write a test for that. The developer then decides, well, is a unit test enough? Do I need an acceptance test?
30:00
But yes, every developer is able to run the whole test suite on their local machine and then run the test. Yes. So, actually like when I say local machine, we have something called developer playground where you log into a machine where you do your actual work on
30:21
and it has like everything ready for you to run the test. So, Docker is installed, everything is there. We have our own local Docker registry. So, it will just, yes, run the tests and you can write them and run them by yourself.
30:44
Hi, thank you very much for the talk. I just have a basic question. How do your services communicate actually? I didn't get this. You mean us in San Francisco? No, your internal services because you said you have a module structure and... Yes. So, it's just HTTP calls with JSON data exchange.
31:04
So, that's basically how they all communicate unless it's like something very special. Thank you.
31:22
You mentioned the acceptance test. When you do deploys, do you do any logging to avoid two deploys of services that are required in the acceptance test? So, if one service depends on service B and both this service and B, they want to be deployed,
31:45
what do you do then? Ah, yeah, okay. That's like, since it's so loosely coupled, either you have to pay like real good attention or you cannot do that. So, each service is considered independent. So, if you need like a deployment
32:02
or you have a deployment of your service and it needs to have some other changes and other services before you can do that deployment, you just, you as a developer, as owner of that service, you need to make sure you don't deploy too early. Usually, like when we do a deploy, the whole test suite is run as the first thing
32:22
even before we go to stage. So, hopefully, if you have good test coverage, you would like notice then that things are not there. But generally, it's your responsibility since we have loose coupling that you don't deploy breaking changes to other services, that you remain backwards compatible and that if you depend on changes from other service,
32:42
that you do the deployment in the right order. Okay, thank you. Hi, thank you for the talk. So, can you please clarify what is the problem with testing services?
33:03
So, you said that each service has a specification. There is some tooling around it like Swagger which verifies that server response matches the specification. And so, there is service A which depends on service B. And why can't just service A expect that service B is always producing valid responses
33:20
and it looks like you don't really need to run a request through all the services around and with the production version of service B for that? Yes, great question. So, it's mainly about the human factor. So, we don't have the tooling in place to check for this automatically. Basically, our acceptance tests are that tooling
33:43
to make sure a new service deployment doesn't break anything. It has happened in the past multiple times. I actually remember once that a service got deployed which then broke something inside Yelp main because Yelp main called that service because the developers, they just didn't think of the fact
34:03
that this small change was actually not backwards compatible. Since we don't have something as rigid to make sure you cannot deploy a change that's not backwards compatible, we have to write tests for that and that's basically our check for that.
34:26
Hi, thanks for the good presentation. If I understood correctly, you have a mobile API like an API specific to mobile applications. Have you ever considered having slightly different structure
34:43
or information written to different types of devices like different for Android and iOS? Yes, we actually do that for what we call consumer apps, so basically the apps you would download on your device.
35:00
We not only might do that depending on device type but also depending on the version of the app you run and other factors, so yes, we do that. Up until now, we have been able to get away with just different version endpoints and sending generally the same data both to Android and iOS
35:21
but we are in the process of actually developing something similar for the BizOwner app as well. Doesn't it make testing your API more complicated? Oh, much more complicated. It's like basically any of these checks is like another branch in your code so yes, that's why we are trying to avoid that as much as possible.
35:44
Yeah, completely agree. Okay, I think one more, okay.
36:01
Hey, thanks for the talk. So, question is, you had a Yelp main and what's the definition of reasons like how you recognize that you need to decouple this main from service? I mean, how you recognize service first? Like, what's the definition of service first?
36:22
Well, we try to put like any new code we write. If it's reasonable, we try to put it inside in service and then use that from Yelp main or wherever to decouple our code. There are also efforts going on in taking code which is already in Yelp main,
36:42
extracting it and putting it into services just so our development speed can increase, ramping up new developers actually becomes much faster since code is just simply less complicated and less huge.
37:04
That's difficult to like answer generally but if you look at our Yelp service principles, there's like actually reasons about that. So, I cannot like mention all of it but I encourage you to check it out. It's actually like handles that topic. And if you want to talk more about this,
37:22
like I will be at the booth now. There's also a bunch of other awesome Yelp engineers there. So, just come talk to us. We're happy to nerd out about this. Thank you again.