Gamers do REST
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 17 | |
| Number of Parts | 119 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/19942 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Place | Berlin |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
EuroPython 201417 / 119
1
2
9
10
11
13
15
17
22
23
24
27
28
41
44
46
49
56
78
79
80
81
84
97
98
99
101
102
104
105
107
109
110
111
112
113
116
118
119
00:00
CodeRepresentational state transferRepresentational state transferLevel (video gaming)Game theoryStatement (computer science)Lecture/Conference
00:43
Game theoryInternet service providerService (economics)Office suiteFront and back endsOrder (biology)Field (computer science)Data managementMoment (mathematics)Right angleElement (mathematics)Computer animation
01:51
Representational state transferStatement (computer science)Wireless LANPosition operatorOffice suiteAuthenticationSlide ruleCountingMobile appGame theoryAuthorizationComputer animationDiagram
03:01
Representational state transferWeb browserScalabilityWebsiteClient (computing)Level (video gaming)Instance (computer science)Mobile appBookmark (World Wide Web)Scaling (geometry)Web pageWeb 2.0Link (knot theory)Software architectureService (economics)Right angleMetadataLecture/Conference
03:55
Representational state transferWeb serviceVideo gameReading (process)Condition numberFormal languageComputing platformMultiplication signCommunications protocolCharge carrierRepresentational state transferVariety (linguistics)Client (computing)CASE <Informatik>Service (economics)Logical constantMultiplicationProcess (computing)Library (computing)Game theoryInstance (computer science)String (computer science)Query languageCodeField (computer science)Computer animation
05:19
RobotMetropolitan area networkClient (computing)Direction (geometry)LogicSuite (music)Multiplication signConstraint (mathematics)Lecture/Conference
05:39
Representational state transferMetropolitan area networkClient (computing)CASE <Informatik>Set (mathematics)CodePlastikkarteLink (knot theory)MappingMultiplication signCommunications protocolRepresentation (politics)Revision controlQuery languageRepresentational state transferScaling (geometry)String (computer science)Service (economics)
06:40
Metropolitan area networkDisintegrationRepresentational state transferRun time (program lifecycle phase)Right angleService (economics)Process (computing)Software testingCycle (graph theory)Physical systemGraphical user interfaceOrder (biology)Lecture/ConferenceComputer animation
07:06
Metropolitan area networkStack (abstract data type)CodePhysical systemHuman migrationTelecommunicationPoint (geometry)Execution unitState of matterMobile WebIntegrated development environmentReal numberMassSubsetData managementBitBuildingEnterprise architectureProduct (business)Multiplication signSparse matrixSoftware developerBranch (computer science)DatabaseProjective planeService (economics)Software repositoryDifferent (Kate Ryan album)Software testingGame theoryScaling (geometry)Lecture/ConferenceComputer animation
09:52
Metropolitan area networkHausdorff spaceArmGrand Unified TheoryMultiplication signRow (database)Arithmetic meanMultiplicationTable (information)Type theorySelectivity (electronic)Software bugLibrary (computing)Configuration spaceRun time (program lifecycle phase)Error message2 (number)Structural loadLoginValidity (statistics)Exception handlingExistential quantificationLevel (video gaming)Process (computing)Computer fileDefault (computer science)MiddlewareGraph (mathematics)Maxima and minimaMetric systemMereologyMobile appSet (mathematics)Operator (mathematics)AuthorizationDependent and independent variablesService (economics)StapeldateiFrequencyClient (computing)LogicPattern matchingScripting languageGame theoryPoint (geometry)Product (business)DebuggerSynchronizationQuicksortSoftware developerMedical imagingWhiteboardLengthFile formatContext awarenessFraction (mathematics)Field (computer science)Form (programming)Data centerSlide ruleCurvatureCartesian coordinate systemString (computer science)Insertion lossFormal verificationAreaSocial classPatch (Unix)Codierung <Programmierung>Semantics (computer science)Source codeDifferent (Kate Ryan album)Student's t-testProjective planeFood energyWritingDescriptive statisticsBlock (periodic table)WordSpacetimeFrame problemCodeIntegerWater vaporEvent horizonFigurate numberCodeConsistencyIntegrated development environmentPhysical systemLecture/Conference
16:08
World Wide Web ConsortiumToken ringHeat transferAutomorphismValue-added networkMetropolitan area networkToken ringDifferent (Kate Ryan album)Public-key cryptographyWeb 2.0LoginComputer wormObject (grammar)Survival analysisEmailOpen setStandard deviationType theoryScaling (geometry)Exception handlingBitClient (computing)Query languageElectronic signatureMultiplication signRepresentational state transferMobile appSoftware frameworkInformation securityAuthenticationContext awarenessCASE <Informatik>Dependent and independent variablesPhysical systemLine (geometry)Revision controlSoftware bugEncryptionMiddlewarePasswordCodeDifferenz <Mathematik>Metric systemAlgorithmView (database)Process (computing)SynchronizationError messageMetropolitan area networkMatrix (mathematics)Response time (technology)Open sourcePosition operatorSource codeCategory of beingConstraint (mathematics)Population densityLibrary (computing)InformationWordSource code
22:08
TriangleSurvival analysisTracing (software)Software frameworkSurfaceProduct (business)Proper mapSoftwareLevel (video gaming)Integrated development environmentVirtual machineRevision controlSoftware testingMultiplication signMereologyAxiom of choiceFrame problemResponse time (technology)Software developerRepresentational state transferStructural loadDifferent (Kate Ryan album)Suite (music)Lecture/Conference
Transcript: English(auto-generated)
00:15
Welcome. It's 3 p.m. and now on stage is Angel Remboi, telling us about RESTful APIs in the gaming industry.
00:35
Hello everyone. Can everyone hear me? So, is that statement accurate? Not really.
00:45
And I'm going to expand on that in just a moment. My name is Angel Remboi and I work for the most awesome company in the world, DemonWare, of course. DemonWare is based in Ireland and when I moved there I had to adapt fast in order to survive in those foreign lands.
01:03
This is my solution. DemonWare is an Activision Blizzard subsidiary. We have offices in Dublin, Vancouver, and Shanghai. We're around 200 people, but we like to keep the startup-ish feel to the company. What do we do?
01:22
Well, basically it can be summed up as straight from one recruiting book. We enable gamers to find one another and then shoot each other in the face. And we're pretty good at it. What we actually do is provide backend services for Activision Game Studios, for leaderboards, matchmaking, anti-cheat, accounts management, and more.
01:44
We have like 70 plus services that serve our past games and also upcoming games like Call of Duty Advanced Warfare and Destiny. And of course we're hiring, so if you're interested in what I'm about to show you, please come talk to me afterwards.
02:05
So back to our previous slide. Is this statement accurate? Well, as you can see from all of our graphs, the user count doesn't come even close to zero. And with over 100 billion API calls per month, well, that's an API's dreamland.
02:25
And these guys get really excited during lunchtime. I mean, I heard HR offices around the world experienced a spike in SIG days requests in November. I want to tell you that this is just a coincidence. It's not us.
02:43
Talk overview. I'm going to touch on topics ranging from API design to authentication and authorization. So let's get to it. Why REST? First, interoperability.
03:01
Our APIs must be available to game clients, websites, companion apps. And by using the right protocol, HTTP in our case, and the REST principles, we can achieve that level of interoperability we need. Second of all, scalability. From all the architectures I've came across, REST looks like the only one that's truly web scale.
03:22
Basically you can look at the web as a huge REST API and your browser is the client that consumes it. So you have your entry point, your bookmark. You use HTTP verbs to talk to web pages. You have links from one page, from one page to another, from one resource to another. You have URIs that define resources and also your browser interprets those resources based on hypertext and metadata.
03:45
So I think it's safe to say that using RESTful architecture style for your API design will make your services easier to scale on the long run.
04:01
Anyone who ever worked in API probably heard of Wright Fielding's thesis. It's an interesting read and when you're done with it, you're left astonished by the elegant concepts outlined in it. You want to adhere to those concepts and you'll probably succeed. But then you realize that whatever you do, your clients will misuse your otherwise perfect creation.
04:25
In the gaming industry, things get more complicated. You have custom protocols because everyone has to do their stuff. You have mandatory libraries and SDKs. You have multiple languages and platforms. Csharp, .net, Java, you name it.
04:40
And the documentation goes from OK to non-existent. Most game developers have little to no contact with web services over the most time of their careers. And some user education sometimes is needed. Even for simple things like what a JSON is or what HTTP codes mean or how to use query strings.
05:05
Only in recent years, the gaming industry started to embrace HTTP and REST-like services that make life easier for us. Having said that, are our APIs RESTful? I want to say yes, but even I have to admit that we don't adhere to all the principles.
05:26
Either because of business constraints, legacy logic or backwards compatibility. The important thing is that we're moving in that direction and we are encouraging our clients to follow suit one step at a time. So design-wise, we get, post, delete verbs for API CRUD.
05:48
We use HTTP for the communication protocol and JSON for representation. And every time we do an API design or work on design, we try to be pragmatic about it. Like good enough is better than perfect.
06:04
Other things we use design-wise, we have versioned URL, but it's mostly semantic. We tend not to break backwards compatibility. It's mostly to tell the client that we have this set of features that it's only available in version 2, for example.
06:21
So we also have a camel case in JSON and query strings. Not in Python code, we have a mapping to underscore for that. We use standardized dates, also links to other resources and of course it's human readable. Just using REST is not enough to run your services at scale.
06:47
You need to have the right process and tools in place. So I'm going to walk you through how we do it at VMware. We use Chrome and Kanban, depending on what works for the team or the registry cycle.
07:02
We also have automatic builds that test everything that's merged into master and tests against also our other systems. So everything is alright, everything to be alright. VMware services use a lot of different techs, so I'm going to focus only on what we use for our APIs.
07:23
So API 7 and Django 1.6. We also use MySQL. At DjangoCon this year, some people were very surprised that we still use MySQL in production, and that scales really well for us. So at one point there was even a show of hands and about 90-95% of the people were using PostgreSQL.
07:46
So I'm curious here at EuroPython, how many people here use MySQL? Can you raise your hands please? PostgreSQL in production? About the same. MongoDB.
08:03
OK. So our reasoning is that we couldn't find enough pros for other SQL databases that would warrant such a big migration for our infrastructure. And MySQL is doing some really good developments in the last few years, so it works pretty well for us.
08:25
Ah, sorry. We also use CentOS 6 and Apache and ModWhiskey. As you can see, we don't use anything flashy. Our layout is simple, reliable, and easily scalable.
08:40
Our projects are built with sharding in mind, also our dev environments and our builds run unit tests, acceptance tests, and all other tests against sharded environments. Our tech stacks, our layout also saves us from a lot of tech stack related issues most of the times, and lets us focus on real business problems.
09:07
Reliability is something we take really seriously, as you can see compared with other big game launches in the past few years. For code, we use Git and GitHub Enterprise. We use feature branches, master is always deployable, we do pull requests for team review, and when
09:27
all the features are in and the builds pass, we bag it, tag it, and ship it. We use RPS for our packaging, individual package of our dependencies, and manage our own repo for dependencies.
09:47
I'm going to talk a bit about schema migrations. So schema migrations are straightforward when you're working with huge amounts of data. So there comes a time when you need to do a schema change, but you cannot afford any downtime.
10:04
And when you have lots and lots of records, an alter can mean table lock. And when you do a table lock, you're going to have a bad time. For this, we use Percona Toolkit, which is a clever set of scripts created by Percona to deal with these kinds of situations.
10:23
We also use Percona MySQL fork in our production environment, but the tool should work for pretty much any MySQL variant. So what does the tool does behind the scenes? It creates an already altered table, a copy of the original table. Then it sets up triggers for insert, update, delete on the old table towards the new table.
10:46
So everything is in sync, for every operation you have consistent data. Then copies the data over in batches, while in the meantime monitors the slave lag, and adjusts the batch size or just stops the operation to let the slave get in sync with master.
11:05
And at the end just renames the table, which is an operation that takes fractions of seconds. The only downside of this process is that it uses a lot of space, as it duplicates all the data in the tables. But other than that, we couldn't find anything in our authors that could deter us from using this tool.
11:28
This is how our configuration file looks like. So why AML? First of all, cross project. We're not a Python only shop, and we need to be consistent with configuration files across the board.
11:45
Also, YAML is just as readable as Python code, and it has to be reviewed by people who don't know Python. Also, validation. We dynamically build the Django settings module at runtime from the loaded YAML file.
12:03
So we check for missing configs and valid values at this point. This way, if something is not right, we know before the actual setting is used. We know when it's loaded, not when it's used in the app. This is a simple example of what our validation library does.
12:25
It checks for type of valid entry. It has default and description, just to give you an idea. On the subject of validation, to validate data sent by our clients, we use JSON schema, and it's pretty awesome.
12:40
If you haven't used it, I really recommend it. As you can see from this example, you can do all kinds of fun stuff with it. You can have restriction by type, minimum maximum integers, pattern matching. You can also have max, minimum length for strings, required fields.
13:02
You can also have different errors depending on what kind of exception is encountered. And the cool part about this is that in your Python code, you can see all the primary validation related to an endpoint or to a resource in one place.
13:21
You don't have to jump hoops or really dig around. Just one thing to mention, we don't use this for heavy business logic validation. It can get messy and it's harder to maintain in the long run.
13:41
For error handling, it's where Django middleware shines. In your views, you just raise the exception in the middleware, we'll catch it in its process exception method. And then you can wrap it in a nice HTTP response and send it back to the client like this. So you might have to do some adjustment depending on what your client needs or what you need.
14:03
But that is basically, we don't do anything fancy there. You can also see we use a hierarchical approach to our errors and also to our error codes. So that's where most of the design work goes into on how to trace back from those errors to the actual thing that happened.
14:26
For logging, syslog relays are logs to an aggregator built on open source tools. In the end, you get something like you see here. We use logstash, select search and kibana for front end.
14:41
These things are somewhat noisy in this slide. But once in a while, you can see exactly that something's happened and you need to act. Of course, we don't spend all day looking at these graphs. We have errors that do that for us. But looking at them, we can have an overview of the frequency of the errors and also the timeframe when that happened.
15:02
For example, deployment, an event, you have a promotion for a game or something like that. We tend to keep the needs of production as opposed to development when we format our logs. The logs need to be concise, complete and contain context. Think about it this way, if you put a log in a bug report, would you understand immediately at first glance where the issue occurred and why?
15:29
So we try to guide us by those principles. A brief example of logging. As you can see, we have the level of the error, the project, the app.
15:40
So we can search pretty much easily based on keywords from all the logs that we get from all our services. Besides our logging, we use metrics, lots of metrics. All our metrics are sent to an aggregator that verifies and sorts them and then sends them to Graphite.
16:04
And you finally get the visual image you see here. So what's the difference between logging and metrics? With metrics, you get different information than with logging. You can have how many mails sent or failed, SQL query times, slave lag over time,
16:22
app response times, user creation over time, user deletion over time. And you notice anomalies like that right away in the middle. Again, Django middleware to the rescue. For this simple example, we record the start time of the request and then in the response,
16:45
we do a diff and we send the request time to our metrics aggregator. It's pretty straightforward. There's nothing, no rocket science there. We can also add here, for example, the process exception method and you have metrics for your exceptions.
17:02
You can log, see metrics of different types of exception over time. For authentication authorization, we use JSON web tokens.
17:26
JSON web tokens contain claims that a system can use to access resources it owns. We use two types of JSON web tokens. We have JSON web signature objects which are claims that are base64 encoded but carry with them a signature for authentication.
17:43
And we also use JSON web encryption objects which are claims that are totally encrypted with a public-private pair. For that, we use a JSON framework created by us in open source. You can find it on GitHub and also pip install it, try it on.
18:01
It's called Jose. I'm going to walk you through how it can be used. As you can see, you have your claims there. You have your issuer, the expired time and the subject and you also have the password. This is for JSON web signature.
18:22
You sign the claims. You can use either a synchronous or a synchronous algorithm for signing your views. Synchronous in this case is a synchronous. This is the object you get. You get header payload and signature. Then you serialize and compact it and send it to the client.
18:43
And the client just like one line of code verifies it and knows it. It's okay. For JSON web encryption objects, it's pretty much the same with the difference that we use private-public key pair.
19:01
And we encrypt it with the public key. You get a slightly bigger object. You serialize it at the client side. You decrypt it with the private key and there you have it. So in summary, REST is awesome.
19:21
Use as many concepts as you can but be pragmatic in your approach. Error logging and metrics monitoring are what makes scalable survival. And scaling is survival. And we're hiring. Of course. And that's it.
19:42
Thank you. Thank you. We have a good five minutes for Q&A. There's one microphone and I'm on this side.
20:02
Hi. Thank you for your talk. Can you talk a little bit more about the last two bits which is JSON web signatures and JSON web encryption? In the sense of how do you guys use it?
20:20
What's the research that you did behind it? Do you believe this is secure? What are the constraints? We did quite some research and we think it's pretty secure for our use case. So for our use cases, I guess it's secure enough.
20:46
We're going to release a new version of HOSI in the near future. We should have more context and better security. Probably this example will be a bit outdated.
21:05
But from our research, as we plan to have some APIs more open in the future, it looks like it's pretty secure.
21:22
I don't know. We can talk about this. We can expand on this subject afterwards. But security is not an easy subject to approach. This is how we do it. We can talk a lot about this. One quick remark. Am I right? JSON web signatures and encryption are open standards, right?
21:42
You did not invent those. Those are open standards? Yes, standards. Open standards. We did not invent... No, no, no. These are based on standards. You wrote the library for that. Sorry. So we did not invent JSON web tokens.
22:03
Hi, thank you for your talk. We briefly discussed this earlier. I was just wondering if there is a specific reason why you decided to implement your own framework rather than using another framework such as a Django framework or another one.
22:22
For the API, you mean? Yes. Well, we use Django, which is a framework in itself, I think. It's safe to say that. At the time when we started developing our own framework, we didn't have a lot of choices, viable choices. Now we have Django REST framework and Tasty Pie, which are pretty mature.
22:46
Back then, I think there was only piston, which is not even maintained anymore. And it didn't suit our needs at that time. So that made us do our own thing.
23:04
Okay, more questions? Hi. Do you use anything for measuring response time or throughput of a new version of your software? You mean like load testing?
23:23
Something like that. Yeah, we do load testing a lot, but it's different. I saw some companies, for example, when they update a new version, they put part of their cluster or a node of their cluster in production and see how it behaves directly in production.
23:40
But we tend not to do that. We load test before with real machines, just with the production-like environment and production-like requests. So the proper staging environment that you test load? Yeah, but it's an exact replica of the production environment usually.
24:04
Okay, then that's it. Thank you very much again.