Building RailsPerf, a toolkit to detect performance regressions in Ruby on Rails core
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 45 | |
| Number of Parts | 94 | |
| Author | ||
| License | CC Attribution - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/30647 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Row (database)Metric systemCodeMobile appMultiplication signObject (grammar)NumberContent (media)Interpreter (computing)String (computer science)Reduction of orderLogical constantMeasurementSemiconductor memoryStatisticsPrimitive (album)File formatDifferenz <Mathematik>Regulärer Ausdruck <Textverarbeitung>QuadrilateralSpeicherbereinigungMultiple RegressionSet (mathematics)Magnetic-core memoryRevision controlInformation technology consultingCache (computing)Group actionProjective planeFunctional (mathematics)State of matterSoftwareAttribute grammarWeb pageRegression analysisStructural loadInterpolationPatch (Unix)Software developerFreezingBlogLetterpress printingVariable (mathematics)Software maintenancePrisoner's dilemmaError message2 (number)DemonType theoryBitSlide rulePoint (geometry)Software frameworkCommutatorUniform resource locatorPattern languageCASE <Informatik>RhombusCentralizer and normalizerOptical disc driveLecture/ConferenceComputer animation
09:54
SpeicherbereinigungMobile appVideo gameStatisticsSystem callBenchmarkIntegrated development environmentDifferent (Kate Ryan album)Stack (abstract data type)Multiplication signTraffic reportingMetric systemSoftware testingAutomationBuildingCodeResultantPrototypeAuthorizationTwitterQuery languageHash functionDatabasePatch (Unix)Magnetic-core memoryRevision controlConnected spaceSemiconductor memoryWeb pageMultilaterationLink (knot theory)Fluid staticsRight angleService (economics)Row (database)Set (mathematics)Software frameworkView (database)Slide ruleMathematicsCartesian coordinate systemProcess (computing)IterationNumberInterpreter (computing)Arithmetic meanTrailProduct (business)Block (periodic table)Mathematical optimizationAlgorithmObject (grammar)Resource allocationConnectivity (graph theory)Electronic mailing listMeasurementBitGroup actionClosed setMereologyEndliche ModelltheorieImage resolutionGame controllerRobotReal numberBit rateOnline helpAreaAdaptive behaviorPerturbation theoryCodeForceComputer configurationForestMarginal distributionEvent horizonComputer animation
19:32
BenchmarkBuildingService (economics)Web applicationRight angleSet (mathematics)Regression analysisServer (computing)Library (computing)Process (computing)Connected spaceCodeResultantRevision controlPoint (geometry)Home pageComputer hardwareView (database)Point cloudMobile appSoftware developerOpen sourceSoftware repositoryInstance (computer science)BefehlsprozessorTrailEmailSuite (music)Object (grammar)AuthorizationCASE <Informatik>Row (database)Remote procedure callPlanningWeb pageMultiplication signProduct (business)InformationComputer architectureConnectivity (graph theory)Computer fileWeb 2.0TwitterWebsiteSelf-organizationLink (knot theory)PrototypeSinc functionMessage passingError messageIntegrated development environmentGroup actionGame controllerVotingSystem callElectronic mailing listTraffic reportingPrimitive (album)1 (number)DatabaseEvent horizonMagnetic-core memoryAutomationLattice (order)Coefficient of determinationWave packetWordRuby on RailsMedical imagingComputer animation
Transcript: English(auto-generated)
00:12
Hello, my name is Kirshetrov. I came from far Russia. I live in Finland now.
00:21
I work on Capistrano as a maintainer, and I'm a Rails committer too. I work at this awesome Rails shop called Evil Martians. We do Rails development, consulting. We developed a group on Russia since the early days.
00:40
We are doing a lot of work for eBay, which is my current project. And my topic today is about performance regressions in Rails. I'll talk about what is a performance regression, how to cache them, how to solve them, what are the tricks to avoid them,
01:03
and about the tool I've been working on last few months to cache performance regressions in Rails automatically, and about the state of this tool and the future.
01:28
Before we start, let's define what is a performance regression. Performance regression is a situation where the software still functions correctly, but it takes more time to compute,
01:41
or it consumes more memory when compared with previous versions. And what is the performance regression in Rails? Imagine a situation where after an upgrade to a new version of Rails, fresh new version, shiny. The page load is increasing from 250 milliseconds
02:03
to four seconds. That's quite huge, right? That's exactly what happened in Rails 3.2.15 because of some bug in sprockets. And now I have a story,
02:21
why we should care about performance regressions in Rails and why is it a problem for all of us. Right after Rails 4.2 release candidate one was released, the discourse team, they have quite a huge app. They decided to upgrade their app on this release candidate one
02:40
just to try Rails 4.2 because they were really excited about all those active record patches by Aaron. They expected some performance boost from adequate record. But what they found, actually, is that this Rails 4.2 release candidate one
03:01
was two times slower than Rails 4.1. That was very surprising. There were some performance regressions, obviously. Luckily, this was just release candidate one and after applying a lot of patches on Rails by discourse team and by Rails team,
03:22
they made Rails 4.2 the last release candidate number three as fast as Rails 4.1 and even a bit faster. Okay, now it seems like a pain in the ass for Rails committers, Rails contributors,
03:42
but not for usual developers who only have some small or bigger Rails app. Why is it a problem for everyone? Because the faster is framework, the faster is your app and by speeding up active record with adequate record patches,
04:02
Aaron Patterson made your apps faster just because active record is usually tied with the typical Rails app. To get a better understanding,
04:26
I have some examples from Rails commits affecting performance and solving the performance regressions. Let's review them. But before we do that, let's think about what kind of metrics do we count on when we talk about performance regressions in Rails.
04:43
These two metrics are timing and allocations. What is timing? Timing is the amount of time, some method or some piece of code takes to execute. By reducing timing, we make our methods faster and we make our app faster.
05:00
Here's an example of code, very small example, how to measure the timing. It's quite straightforward. And we want to reduce number of allocated objects in Ruby interpreter memory because garbage collection is quite an expensive operation in Ruby and by allocating less objects, we leave less work for garbage collector.
05:22
Here is an example how to measure number of objects allocated in your code. We get statistics from garbage collector and then calculate it. And most of, a lot of performance issues can be solved
05:41
just by keeping in mind a set of tricks. How do basic data primitives in Ruby work? By these basic primitives, I mean strings, hashes, blocks, arrays, and so on. So we will start from comments from Rails that are working with strings,
06:03
having some issues with strings. First comment by Aaron Patterson. Now I zoomed the diff. So the problem of this code was that it allocated, okay, the problem of this code was that it used plus method to concatenate to strings
06:22
while, in fact, interpolation is way faster, really way faster. Another problem this thing solved is that we don't allocate one more string before we concatenate them. We just use the interpolation once.
06:41
Here's another example by Sam Saffron, who is actually in this room. So the problem was that this code was called on every active record attribute read. Imagine that. Reading attributes is the core functionality
07:01
of the active record, and on every attribute read, we allocated a new string. And by moving out the string into the constant freezing this constant, Sam reduced a number of allocated objects on the discourse front app by 5,000. That's 5,000 objects
07:20
just by moving out one constant, one string to the constant. Here is another comment. It's not from Rails itself. It's from the Rack gem.
07:41
It's not Rails the euro, but as you know, Rack gem is used by Rails a lot. So the problem of this piece of code was that it used... On every request, it allocated new variables, new strings, like content type, content length, and there were many of them.
08:01
So on every new request, we allocated a bunch of new strings and then used them. But using the same trick I described in the previous slide, moving out these strings into constant reduced number of allocated objects on one request to Rack by 40%.
08:24
40%. That's almost a half of all objects inside the gem. We reduced it just by moving out this constant. This was quite a popular pull request on GitHub. Another example. It is from Active Support.
08:40
It is... It's working with time formatting. And just to replace the time pattern, we used Gsub four times, allocated a new string every time. But in fact, we could use only one regex
09:01
to regular expression to do all the same things. So we don't need four Gsubs, right? Just one regular expression. And this is the commit by my colleague, Ravel. His nickname is Brainopia. The thing was that in Ruby 2.2,
09:22
in previous versions of Rails, we used regular expressions for string replacement, like this, because replacing with regular expression was faster in Ruby 2.1. But it's an interesting case, because in Ruby 2.2,
09:43
replacing strings with strings is now faster. 1.5 times faster. And this quad string method, it is called on every, on every, when you make a new query to your database.
10:01
So if you have, let's say, 100 queries on some app, it's called 100 times. And now we have some examples working with hashes. Here is a piece of code from URL4 helper.
10:21
And to override just one hash options, we allocated new array and code merge just to override one hash value. But as you can see, we can just use the hash set method without allocating new array, without using merge. And here is the same trick by Aaron Patterson.
10:43
We don't need to, as I said, we don't need to allocate new hash. We just set the value of the same hash without merge. Just right. It's here. So Aaron has done a lot of search performance patches in ActiveRecord,
11:02
and he's in a great job. And one more example, well, about working with hashes. This piece of code is from ActiveRecord connection adapters,
11:21
and it was called from there. And when we wanted to iterate over all hash values, we allocated new array with all hash values by using the values method, and then we iterated over it. But if we think there is a method called each value to do the same thing,
11:43
and by using this method, we can avoid allocating new array in memory. We can, and this array may be quite big if the hash is big. It's good to remember that we want to optimize only hot code. By hot code, I mean the code,
12:02
which is called frequently on every request, many times on every request. And Rails code is, like, code in the Rails, in the framework, is usually, all the code in Rails is hot because it's framework used by thousands and thousands of different Rails applications.
12:22
And there is, it makes no sense to optimize the piece of code, which is called once a month from a background job. You won't benefit much, right? And now you have some idea what kind of performance changes I mean. Here in my talk. And what if we track these changes
12:42
to see the Rails performance, the overall Rails performance? This idea to build a service to track Rails performance changes was suggested in the Rails base camp, and I became really interested because it's a great idea how we can optimize,
13:01
how we can save time for Rails contributors. Obviously, to track these changes, we need some kind of benchmarks to benchmark the code. Are there any existing benchmarks in Rails? The answer is yes. There is great benchmarks right inside the discourse application.
13:22
This is not a Rails benchmark. This is just a benchmark of discourse, and discourse app is using Rails, the framework. So in this, in this benchmark, discourse is setting up the database, populating these database with records, starting unicorn in production environment,
13:42
and making a lot, a lot of requests using Apache Bench. And then this benchmark brings timing and memory usage. It's a great benchmark. It is used, it was used even by Koichi to optimize garbage collection algorithms in MRI.
14:02
And this benchmark is used every time discourse upgrades on new Rails version to see if this Rails version is faster or not. It's a great approach, as I said, but we also want to measure performance of Rails components separately.
14:22
I started from making a list of these components. You can see it on my slide. I didn't include active support here because active support is something very stable. We don't change it much, and it's almost frozen. And some parts we change a lot.
14:41
Active record, action controller, action view. I wrote some benchmarks. I run them against a few major Rails versions. You can see these versions on my slides. And... Just a second.
15:06
And I visualized these benchmarks on the static page using D3 JavaScript framework. I will give a link to this static page later.
15:24
You can see these benchmarks. The higher is the bar, the better is performance. We use iteration per second here to measure. You can see the trend, how Rails becomes faster and faster, right? Now I will talk about what these benchmarks are doing.
15:41
So here is an example of benchmark of active record finders. We create one record. We create one record in the database, the user record, and then we are trying to find this record using the find method. We're trying to find it actually 100 times.
16:01
This is what we call microbenchmark. You can see some special helper called benchmark.trails. And let's check out what this helper is doing inside. So this helper disables garbage collector if it's necessary. It warms up the code because it's really important to warm up the code,
16:21
to warm up the interpreter before we run our benchmarks. Then it makes some number of iterations. This number can be 100 or 1,000 or a few thousand. And then we calculate the mean timing and object allocations. Those two metrics I mentioned before.
16:44
Here is another example of the benchmark of create method in active record. We create one record inside the benchmark block and then see the result. Now I have a story. Why is it so important to have a core benchmark
17:02
and to run them in the production, in the environment close to production? Here is the full app benchmark. By full app, I mean some small Rails app with few controllers, few models.
17:22
And we, in this benchmark, we are making a few requests to this app. Here is the final version. And we can see that Rails 4.2 is a bit faster than some previous versions. But the first version of this benchmark looked like this. Obviously, I didn't believe it
17:41
because it was some mistake. And I decided to discover what's happening inside with a tool called StackProf. This is still made by Amman Gupta. This is a tool to see calls inside the Ruby stack and see some statistics about these calls.
18:04
Here is an example of difference between stack calls in 4.1 and 4.2. What I discovered is that the environment of this app that I run was not close to production environment that we have in Rails. And after fixing it,
18:25
the benchmarks started to look closer to the real life. Now we have some set of benchmarks and it's time to automate running them against new versions of Rails. We love automating things in Ruby community.
18:44
There are great examples of automating. Examples like Travis CI that automates build metrics and tests and Hound CI that automates code review and automates review of the code for Ruby guidelines.
19:01
So we can build some service to automate running these benchmarks against new versions, new comments of Rails. What do we want from this service? Just as I said, run benchmark for every new comment in Rails, run benchmark for every new pull request, and report results of this benchmark
19:21
to the pull request author. And to see some trends in Rails to see the charts. I started from making a prototype of the service. I built it in January. It was very small and simple. It could make some things like fetching a new comment
19:40
from Rails and running only two benchmarks against it because in January it had only two benchmarks. Now we have more. At some point I felt like I'm building yet another Travis CI with all of those infrastructure and a lot of code, just like reinventing the wheel. And just a month before I started development
20:03
there was a service called RubyBench made by Sam Saffron and Alan. Please raise your hand if you heard about RubyBench. Okay, nice. For those of us who don't know about RubyBench yet, RubyBench is a service to automate running benchmarks
20:23
for Ruby, for MRI. So in every comment to MRI repo on GitHub it runs new benchmarks. It has nice infrastructure powered with Docker and it has sponsored hardware because it's extremely important to run your benchmarks
20:44
on a dedicated hardware because if you run these benchmarks in cloud the CPU in cloud is shared between many instances and if someone else is mining Bitcoin on another instance it may affect your benchmark results, which is bad.
21:03
And after getting in contact with RubyBench authors we decided that I will just embed Rails support into RubyBench. And now let's see how it works. On every new comment to the GitHub repo,
21:21
on this example it's Ruby slash Ruby repo. GitHub makes a webhook request to our web application. This web application is starting a background job with remote connection to our Docker server and where the Docker container is created
21:42
with this exact version of Ruby with the benchmark suite for Ruby or for Rails, depends on the repo. And right after the benchmark suite is finished the result is reported back to the web app where we can see the updated chart.
22:02
Here is the UI of RubyBench and there is a, you can choose the benchmark on the sidebar on the left and on the right you will see the chart for this benchmark, you can zoom it, you can check out the comments and so on.
22:22
Now I have some information about the architecture of RubyBench. Obviously we have three components.
22:40
You can find all those three components on GitHub. There is a web application, there is a repo with Docker files and there is a repo with a benchmark suite. Why do we use Docker here? Because Docker is a great way to isolate environments, to isolate dependencies and otherwise it would be
23:01
just messy to keep all those trunk rubies built inside one RBM or RBM user. We have a benchmark suite repo with a lot of microbenchmarks for Rails for Ruby.
23:21
And we update the benchmarks use sometimes but now there are like 100, maybe less, benchmarks, microbenchmarks for Ruby itself and about from 10 to 20 benchmarks for Rails and we're working on adding new Rails benchmarks.
23:40
Now we benchmark ActiveRecord, ActionController, ActionDispatch, Rooter and Views. What about the BAP app? It's just a simple Rails 4.2 app run on Heroku. We're using delayed job to run a remote connection to our Docker server in background
24:01
and we use HighCharts JavaScript library to render all those nice charts. Let's talk about results of RubyBench and about Rails support there. So now we have a set of benchmarks for Rails. We run builds for Rails for every new version of Rails. We are working on support on per commit benchmarks.
24:24
I think we will launch it in next few days. What is exciting is that since November there are already 250,000 builds for Ruby, builds of microbenchmarks, and what's even more exciting is that there was already one performance regression
24:42
of the Ruby code and this performance regression could be solved on the next day after the commit, after the run commit was made because we knew exactly which comment caused this performance regression. What are our plans about benchmarking Ruby and Rails?
25:04
We want to implement the pull request benchmarks for every new pull request to Rails just to report that the performance back to the pull request author. We also want to make weekly reports and in this case some contributors can get an email
25:21
like in this week, actual record became five percent faster or maybe not. If the performance remains the same we simply don't need to send this email and we also want to implement your Ruby support because we already have a set of benchmarks for Ruby
25:40
and we can just run the same benchmarks against JRuby. So today we talked about what is a performance regression, how they look like in Rails, how they can be solved, how to find them, how to track them, what are the tools. We talked about building a benchmark suite
26:02
and automating the tracking of the performance. But wait, what if you're not a Rails contributor and you're not planning to be ever? How can all these tricks help you to make your app faster and to avoid performance regression and just to avoid performance regression, right?
26:24
To track your own app performance. First you can start by studying how do Ruby primitives, basic Ruby objects work and how to treat them properly. There is a great book called Ruby Under Microscope by Pat Shaughnessy.
26:41
I mention this book in my talk because it covers all the data primitives, how do they work, not only in MRI but also in Rubinius in JRuby. If you're interested in more tricks like I showed about writing fast Ruby,
27:01
you can check out this talk by Eric Michael Sober. It's called Writing Fast Ruby. And so next you can write a discourse-like benchmark for your own Rails app. This benchmark can set up the application, set up the database, populate it with some records,
27:22
start a server in production environment, make a bunch of requests with Apache Bench or some other tool to some list of endpoints that are important for you, like the index page or some other page. And then print timings and just run this benchmark
27:41
after every comment on Travis and have some idea of the performance. You can also subscribe to the Rails Weekly mailing list. It's also called This Week in Rails. In this mailing list, Rails contributors write about what happened in the Rails repo on this week,
28:02
what are new features, or if we fixed some regression in Rails or fixed some issue. Here are some links about my talk. So the first one is about the static page with all those Rails benchmarks
28:21
since Rails 3.2 and until Rails 4.2. Another one is the initial prototype of the service to benchmark Rails. I call it Railsperf. It's an open source on my GitHub. And you can also check out RubyBench components
28:42
on GitHub in our organization. Here are my usernames on Twitter and GitHub. Please follow me. Also please don't hesitate to contact me if you are interested in this topic about Rails benchmarks, about RubyBench.
29:02
If you want to contribute somehow, we will be more than happy to discuss it. You can also check out the Twitter and website of my company. And thank you so much for coming to this talk.